[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:26:21 dist-test-slave-qk05 krb5kdc[24171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:26:21 dist-test-slave-qk05 krb5kdc[24171](info): set up 2 sockets
Sep 29 11:26:21 dist-test-slave-qk05 krb5kdc[24171](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 11:26:22.232281 24164 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:26:22 dist-test-slave-qk05 krb5kdc[24171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:26:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:26:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:26:22.415118 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:33099
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:39183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:33099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:22.768345 24184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:22.768719 24184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:22.768848 24184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:22.777148 24184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:22.777330 24184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:22.777400 24184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:22.777474 24184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:22.777535 24184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:22.788112 24184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:39183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:33099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:33099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:22.789561 24184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:22.790956 24184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:22.804629 24189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:22.804591 24190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:22.804539 24192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:22.805462 24184 server_base.cc:1034] running on GCE node
I20260929 11:26:22.806342 24184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:22.808049 24184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:22.809415 24184 hybrid_clock.cc:648] HybridClock initialized: now 1790681182809315 us; error 55 us; skew 500 ppm
I20260929 11:26:22.810141 24184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:22 dist-test-slave-qk05 krb5kdc[24171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681182, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:22.814661 24184 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:26:22.816534 24184 webserver.cc:460] Webserver started at http://127.23.153.62:36743/ using document root <none> and password file <none>
I20260929 11:26:22.817320 24184 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:22.817441 24184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:22.817771 24184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:22.821095 24184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "00465b12db5c489883be0746314d6fca"
format_stamp: "Formatted at 2026-09-29 11:26:22 on dist-test-slave-qk05"
server_key: "2cb4de79188cfd4e6c1a9b1bcdbf6ad7"
server_key_iv: "97f0baf4388dfa514a4ba65b466a7417"
server_key_version: "encryptionkey@0"
I20260929 11:26:22.821996 24184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "00465b12db5c489883be0746314d6fca"
format_stamp: "Formatted at 2026-09-29 11:26:22 on dist-test-slave-qk05"
server_key: "2cb4de79188cfd4e6c1a9b1bcdbf6ad7"
server_key_iv: "97f0baf4388dfa514a4ba65b466a7417"
server_key_version: "encryptionkey@0"
I20260929 11:26:22.827941 24184 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 11:26:22.832023 24199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:22.833643 24184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:26:22.833854 24184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "00465b12db5c489883be0746314d6fca"
format_stamp: "Formatted at 2026-09-29 11:26:22 on dist-test-slave-qk05"
server_key: "2cb4de79188cfd4e6c1a9b1bcdbf6ad7"
server_key_iv: "97f0baf4388dfa514a4ba65b466a7417"
server_key_version: "encryptionkey@0"
I20260929 11:26:22.834079 24184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:22.855448 24184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:22.860446 24184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:22.860764 24184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:22.880782 24184 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:33099
I20260929 11:26:22.880807 24250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:33099 every 8 connection(s)
I20260929 11:26:22.882266 24184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:26:22.883687 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 24184
I20260929 11:26:22.883970 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:26:22.886193 24164 external_mini_cluster.cc:1428] Setting key 069ef45332a6d7644630b131e79540fd
I20260929 11:26:22.886502 24251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:22.901302 24251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca: Bootstrap starting.
I20260929 11:26:22.906668 24251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:22.908236 24251 log.cc:824] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:22.911828 24251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca: No bootstrap required, opened a new log
Sep 29 11:26:22 dist-test-slave-qk05 krb5kdc[24171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:22.922385 24251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00465b12db5c489883be0746314d6fca" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33099 } }
I20260929 11:26:22.922827 24251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:22.922930 24251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00465b12db5c489883be0746314d6fca, State: Initialized, Role: FOLLOWER
I20260929 11:26:22.924027 24251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00465b12db5c489883be0746314d6fca" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33099 } }
I20260929 11:26:22.924656 24256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:26:22.924950 24256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00465b12db5c489883be0746314d6fca" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33099 } }
I20260929 11:26:22.924629 24251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:26:22.927418 24251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:26:22.927600 24251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:22.927508 24256 leader_election.cc:302] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00465b12db5c489883be0746314d6fca; no voters:
I20260929 11:26:22.928424 24256 leader_election.cc:288] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260929 11:26:22.931550 24251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00465b12db5c489883be0746314d6fca" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33099 } }
I20260929 11:26:22.931895 24251 leader_election.cc:302] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00465b12db5c489883be0746314d6fca; no voters:
I20260929 11:26:22.931968 24257 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260929 11:26:22.932164 24251 leader_election.cc:288] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:26:22.932662 24257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:22.932860 24257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [term 1 LEADER]: Becoming Leader. State: Replica: 00465b12db5c489883be0746314d6fca, State: Running, Role: LEADER
I20260929 11:26:22.933771 24257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00465b12db5c489883be0746314d6fca" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33099 } }
I20260929 11:26:22.934527 24251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:26:22.940317 24256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00465b12db5c489883be0746314d6fca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00465b12db5c489883be0746314d6fca" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33099 } } }
I20260929 11:26:22.940346 24258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00465b12db5c489883be0746314d6fca. Latest consensus state: current_term: 1 leader_uuid: "00465b12db5c489883be0746314d6fca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00465b12db5c489883be0746314d6fca" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33099 } } }
I20260929 11:26:22.940703 24256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:22.940750 24258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:22.942480 24262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:26:22.942504 24253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:22.890684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.23.153.62:33099)
0929 11:26:22.891835 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:22.891859 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:22.893367 (+ 1508us) server_negotiation.cc:407] Connection header received
0929 11:26:22.905665 (+ 12298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:22.905694 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:22.906194 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:22.906591 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:22.910902 (+ 4311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:22.912522 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:22.915357 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:22.916223 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:22.916428 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:22.925009 (+ 8581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:22.925018 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:22.925039 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:22.925113 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:22.930570 (+ 5457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:22.931900 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:22.931911 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:22.931933 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:22.932246 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:22.933017 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:22.933022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:22.933026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:22.934108 (+ 1082us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:22.934458 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:22.938291 (+ 3833us) server_negotiation.cc:299] Negotiation successful
0929 11:26:22.939275 (+ 984us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":487,"threads_started":1}
I20260929 11:26:22.949229 24262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:26:22.962321 24262 catalog_manager.cc:1347] Generated new cluster ID: 031ff320af4545119869a1936ecea07c
I20260929 11:26:22.962466 24262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:26:22.979328 24262 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:26:22.981427 24262 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:26:23.001580 24262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00465b12db5c489883be0746314d6fca: Generated new TSK 0
I20260929 11:26:23.002749 24262 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
I20260929 11:26:23.056653 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:33099
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:39183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:23.434648 24277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:23.434988 24277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:23.435133 24277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:23.443212 24277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:23.443341 24277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:23.443472 24277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:26:23.454138 24277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:39183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:33099
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:23.455891 24277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:23.458540 24277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:23.477073 24283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:23.478412 24282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:23.478596 24285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:23.480756 24277 server_base.cc:1034] running on GCE node
I20260929 11:26:23.481860 24277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:23.483762 24277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:23.485383 24277 hybrid_clock.cc:648] HybridClock initialized: now 1790681183485296 us; error 84 us; skew 500 ppm
I20260929 11:26:23.485915 24277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:23 dist-test-slave-qk05 krb5kdc[24171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681183, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:23.491799 24277 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:26:23.494482 24277 webserver.cc:460] Webserver started at http://127.23.153.1:39807/ using document root <none> and password file <none>
I20260929 11:26:23.495628 24277 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:23.495805 24277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:23.496241 24277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:23.501031 24277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "ec27c1d727334787a8b66e2e839e6e26"
format_stamp: "Formatted at 2026-09-29 11:26:23 on dist-test-slave-qk05"
server_key: "2e564c744cd65eba9f7baa9eaf0638c1"
server_key_iv: "6e4d7cdf00850f70373886d6d555188c"
server_key_version: "encryptionkey@0"
I20260929 11:26:23.502573 24277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "ec27c1d727334787a8b66e2e839e6e26"
format_stamp: "Formatted at 2026-09-29 11:26:23 on dist-test-slave-qk05"
server_key: "2e564c744cd65eba9f7baa9eaf0638c1"
server_key_iv: "6e4d7cdf00850f70373886d6d555188c"
server_key_version: "encryptionkey@0"
I20260929 11:26:23.513217 24277 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.009s sys 0.000s
I20260929 11:26:23.517872 24292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:23.519706 24277 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 11:26:23.519920 24277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "ec27c1d727334787a8b66e2e839e6e26"
format_stamp: "Formatted at 2026-09-29 11:26:23 on dist-test-slave-qk05"
server_key: "2e564c744cd65eba9f7baa9eaf0638c1"
server_key_iv: "6e4d7cdf00850f70373886d6d555188c"
server_key_version: "encryptionkey@0"
I20260929 11:26:23.520161 24277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:23.535655 24277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:23.540593 24277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:23.540952 24277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:23.542081 24277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:26:23.543726 24277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:23.543828 24277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:23.543979 24277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:23.544040 24277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:23.578576 24277 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:33667
I20260929 11:26:23.578608 24404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:33667 every 8 connection(s)
I20260929 11:26:23.579993 24277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:26:23.587744 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 24277
I20260929 11:26:23.587981 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:26:23.588721 24164 external_mini_cluster.cc:1428] Setting key 047c665e66fc7490b55180b4852c12eb
Sep 29 11:26:23 dist-test-slave-qk05 krb5kdc[24171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681183, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.153.2@KRBTEST.COM; defaulting to no policy
I20260929 11:26:23.605459 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:23.582857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:46517 (local address 127.23.153.62:33099)
0929 11:26:23.583377 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:23.583381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:23.584491 (+ 1110us) server_negotiation.cc:407] Connection header received
0929 11:26:23.586746 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:23.586753 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:23.586861 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:23.586963 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:23.588828 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:23.589698 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:23.592071 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:23.592523 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:23.592653 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:23.597358 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:23.597365 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:23.597368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:23.597413 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:23.600893 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:23.601832 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:23.601836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:23.601839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:23.601976 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:23.602455 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:23.602459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:23.602461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:23.602770 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:23.602928 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:23.604573 (+ 1645us) server_negotiation.cc:299] Negotiation successful
0929 11:26:23.604846 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":282,"threads_started":1}
I20260929 11:26:23.606669 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:23.583463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:33099 (local address 127.23.153.1:46517)
0929 11:26:23.584279 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:23.584328 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:23.586357 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:23.587296 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:23.587308 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:23.587896 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:23.588607 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:23.588629 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:23.589894 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:23.589901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:23.591862 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:23.591883 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:23.592803 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:23.592812 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:23.593089 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:23.594320 (+ 1231us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:23.594353 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:23.596933 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:23.601114 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:23.601131 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:23.601148 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:23.601674 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:23.602098 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:23.602102 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:23.602104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:23.602324 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:23.602908 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:23.602918 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:23.603730 (+ 812us) client_negotiation.cc:769] Sending connection context
0929 11:26:23.604367 (+ 637us) client_negotiation.cc:240] Negotiation successful
0929 11:26:23.605045 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":300,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.2@KRBTEST.COM" created.
I20260929 11:26:23.608512 24405 heartbeater.cc:347] Connected to a master server at 127.23.153.62:33099
I20260929 11:26:23.608935 24405 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:23.610105 24405 heartbeater.cc:511] Master 127.23.153.62:33099 requested a full tablet report, sending...
I20260929 11:26:23.613940 24216 ts_manager.cc:194] Registered new tserver with Master: ec27c1d727334787a8b66e2e839e6e26 (127.23.153.1:33667)
I20260929 11:26:23.615787 24216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:46517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
I20260929 11:26:23.635567 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:33099
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:39183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:24.015300 24411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:24.015655 24411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:24.015810 24411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:24.023914 24411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:24.024099 24411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:24.024260 24411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:26:24.034776 24411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:39183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:33099
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:24.036244 24411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:24.037635 24411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:24.051772 24419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:24.052150 24417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:24.051833 24416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:24.057502 24411 server_base.cc:1034] running on GCE node
I20260929 11:26:24.058249 24411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:24.059403 24411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:24.060638 24411 hybrid_clock.cc:648] HybridClock initialized: now 1790681184060587 us; error 49 us; skew 500 ppm
I20260929 11:26:24.060987 24411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:24 dist-test-slave-qk05 krb5kdc[24171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681184, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:24.067883 24411 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:26:24.069972 24411 webserver.cc:460] Webserver started at http://127.23.153.2:33617/ using document root <none> and password file <none>
I20260929 11:26:24.070899 24411 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:24.071153 24411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:24.072005 24411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:24.076118 24411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "1ac0413c3dd04431b427f49591777b82"
format_stamp: "Formatted at 2026-09-29 11:26:24 on dist-test-slave-qk05"
server_key: "ca8ae7d8b51f619231f85bfb0ee7b6b2"
server_key_iv: "9827a215295147e8a8fc97a856f37c2c"
server_key_version: "encryptionkey@0"
I20260929 11:26:24.077059 24411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "1ac0413c3dd04431b427f49591777b82"
format_stamp: "Formatted at 2026-09-29 11:26:24 on dist-test-slave-qk05"
server_key: "ca8ae7d8b51f619231f85bfb0ee7b6b2"
server_key_iv: "9827a215295147e8a8fc97a856f37c2c"
server_key_version: "encryptionkey@0"
I20260929 11:26:24.083063 24411 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260929 11:26:24.087455 24426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:24.089075 24411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:26:24.089291 24411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "1ac0413c3dd04431b427f49591777b82"
format_stamp: "Formatted at 2026-09-29 11:26:24 on dist-test-slave-qk05"
server_key: "ca8ae7d8b51f619231f85bfb0ee7b6b2"
server_key_iv: "9827a215295147e8a8fc97a856f37c2c"
server_key_version: "encryptionkey@0"
I20260929 11:26:24.089517 24411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:24.118306 24411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:24.122924 24411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:24.123265 24411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:24.124377 24411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:26:24.125952 24411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:24.126037 24411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:24.126157 24411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:24.126210 24411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:24.160488 24411 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:42611
I20260929 11:26:24.160531 24538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:42611 every 8 connection(s)
I20260929 11:26:24.161823 24411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:26:24.166456 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 24411
I20260929 11:26:24.166635 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:26:24.167066 24164 external_mini_cluster.cc:1428] Setting key e0a0cdf29f354bb81bd271d124cd9c98
Sep 29 11:26:24 dist-test-slave-qk05 krb5kdc[24171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681184, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.3@KRBTEST.COM" created.
I20260929 11:26:24.186872 24541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:24.164546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:45317 (local address 127.23.153.62:33099)
0929 11:26:24.164992 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:24.164997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:24.166038 (+ 1041us) server_negotiation.cc:407] Connection header received
0929 11:26:24.168334 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:24.168339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:24.168416 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:24.168532 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:24.171402 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:24.172283 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:24.174798 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:24.175487 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:24.175641 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:24.179635 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:24.179644 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:24.179651 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:24.179706 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:24.183172 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:24.184202 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:24.184208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:24.184210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:24.184379 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:24.184880 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:24.184892 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:24.184894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:24.185255 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:24.185398 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:24.186301 (+ 903us) server_negotiation.cc:299] Negotiation successful
0929 11:26:24.186555 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":210,"threads_started":1}
I20260929 11:26:24.188246 24542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:24.165131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:33099 (local address 127.23.153.2:45317)
0929 11:26:24.165854 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:24.165894 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:24.167905 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:24.168840 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:24.168854 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:24.170071 (+ 1217us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:24.171045 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:24.171080 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:24.172481 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:24.172487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:24.174574 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:24.174594 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:24.175781 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:24.175788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:24.176028 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:24.176754 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:24.176781 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:24.179266 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:24.183388 (+ 4122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:24.183398 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:24.183416 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:24.183976 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:24.184498 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:24.184502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:24.184504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:24.184757 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:24.185430 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:24.185438 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:24.185869 (+ 431us) client_negotiation.cc:769] Sending connection context
0929 11:26:24.186144 (+ 275us) client_negotiation.cc:240] Negotiation successful
0929 11:26:24.186683 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":277,"threads_started":1}
I20260929 11:26:24.189956 24539 heartbeater.cc:347] Connected to a master server at 127.23.153.62:33099
I20260929 11:26:24.190305 24539 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:24.191200 24539 heartbeater.cc:511] Master 127.23.153.62:33099 requested a full tablet report, sending...
I20260929 11:26:24.193207 24216 ts_manager.cc:194] Registered new tserver with Master: 1ac0413c3dd04431b427f49591777b82 (127.23.153.2:42611)
I20260929 11:26:24.194139 24216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:45317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
I20260929 11:26:24.209954 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:33099
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:39183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:24.553686 24545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:24.554037 24545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:24.554337 24545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:24.562449 24545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:24.562625 24545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:24.562786 24545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:26:24.574216 24545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:39183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:33099
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:24.576215 24545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:24.577718 24545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:24.592737 24550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:24.592988 24553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:24.593204 24551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:24.594676 24545 server_base.cc:1034] running on GCE node
I20260929 11:26:24.595425 24545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:24.596509 24545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:24.597746 24545 hybrid_clock.cc:648] HybridClock initialized: now 1790681184597695 us; error 52 us; skew 500 ppm
I20260929 11:26:24.598110 24545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:24 dist-test-slave-qk05 krb5kdc[24171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681184, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:24.602041 24545 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:26:24.603931 24545 webserver.cc:460] Webserver started at http://127.23.153.3:33447/ using document root <none> and password file <none>
I20260929 11:26:24.604712 24545 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:24.604840 24545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:24.605165 24545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:24.608537 24545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "cbd15227791d4b528c4d2b6bea8e2260"
format_stamp: "Formatted at 2026-09-29 11:26:24 on dist-test-slave-qk05"
server_key: "028613b9985e5e0cb4b01f8da66deeb9"
server_key_iv: "619f976f9fab3925bf487a25b85b84ec"
server_key_version: "encryptionkey@0"
I20260929 11:26:24.609427 24545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "cbd15227791d4b528c4d2b6bea8e2260"
format_stamp: "Formatted at 2026-09-29 11:26:24 on dist-test-slave-qk05"
server_key: "028613b9985e5e0cb4b01f8da66deeb9"
server_key_iv: "619f976f9fab3925bf487a25b85b84ec"
server_key_version: "encryptionkey@0"
I20260929 11:26:24.615911 24545 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 11:26:24.621326 24405 heartbeater.cc:503] Master 127.23.153.62:33099 was elected leader, sending a full tablet report...
I20260929 11:26:24.621596 24560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:24.624434 24545 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 11:26:24.624665 24545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "cbd15227791d4b528c4d2b6bea8e2260"
format_stamp: "Formatted at 2026-09-29 11:26:24 on dist-test-slave-qk05"
server_key: "028613b9985e5e0cb4b01f8da66deeb9"
server_key_iv: "619f976f9fab3925bf487a25b85b84ec"
server_key_version: "encryptionkey@0"
I20260929 11:26:24.624892 24545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:24.641276 24545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:24.646993 24545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:24.647367 24545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:24.648478 24545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:26:24.649855 24545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:24.649940 24545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:24.650043 24545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:24.650089 24545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:24.683538 24545 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:45965
I20260929 11:26:24.683588 24672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:45965 every 8 connection(s)
I20260929 11:26:24.684834 24545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:26:24.687616 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 24545
I20260929 11:26:24.687820 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:26:24.688498 24164 external_mini_cluster.cc:1428] Setting key 28ac3993b27474269e9a35a78c47c493
Sep 29 11:26:24 dist-test-slave-qk05 krb5kdc[24171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681184, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:24.707563 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:24.687603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:34211 (local address 127.23.153.62:33099)
0929 11:26:24.691416 (+ 3813us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:24.691422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:24.691442 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:26:24.692329 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:24.692335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:24.692444 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:24.692538 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:24.694616 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:24.695508 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:24.697126 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:24.697512 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:24.697622 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:24.701255 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:24.701261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:24.701275 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:24.701338 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:24.704125 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:24.705130 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:24.705136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:24.705140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:24.705303 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:24.705861 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:24.705865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:24.705867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:24.706166 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:24.706289 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:24.707102 (+ 813us) server_negotiation.cc:299] Negotiation successful
0929 11:26:24.707292 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3687,"thread_start_us":3577,"threads_started":1}
I20260929 11:26:24.708734 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:24.688445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:33099 (local address 127.23.153.3:34211)
0929 11:26:24.689744 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:24.689808 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:24.692010 (+ 2202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:24.692851 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:24.692864 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:24.693451 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:24.694313 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:24.694396 (+ 83us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:24.695668 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:24.695672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:24.696972 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:24.696986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:24.697736 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:24.697743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:24.697982 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:24.698642 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:24.698665 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:24.700975 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:24.704312 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:24.704324 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:24.704347 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:24.704991 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:24.705461 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:24.705467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:24.705471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:24.705742 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:24.706310 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:24.706317 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:24.706711 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 11:26:24.706973 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 11:26:24.707559 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":654,"threads_started":1}
I20260929 11:26:24.710282 24673 heartbeater.cc:347] Connected to a master server at 127.23.153.62:33099
I20260929 11:26:24.710601 24673 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:24.711407 24673 heartbeater.cc:511] Master 127.23.153.62:33099 requested a full tablet report, sending...
I20260929 11:26:24.713109 24216 ts_manager.cc:194] Registered new tserver with Master: cbd15227791d4b528c4d2b6bea8e2260 (127.23.153.3:45965)
I20260929 11:26:24.713984 24216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:34211
I20260929 11:26:24.718546 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 11:26:24 dist-test-slave-qk05 krb5kdc[24171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681184, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 11:26:24 dist-test-slave-qk05 krb5kdc[24171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681184, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:26:24.764894 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:24.747456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.23.153.1:33667)
0929 11:26:24.748041 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:24.748049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:24.748528 (+ 479us) server_negotiation.cc:407] Connection header received
0929 11:26:24.748753 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:24.748761 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:24.748983 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:24.749131 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:24.750551 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:24.751717 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:24.753293 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:24.754219 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:24.754342 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:24.758690 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:24.758708 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:24.758726 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:24.758768 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:24.761777 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:24.762568 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:24.762576 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:24.762578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:24.762722 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:24.763273 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:24.763278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:24.763280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:24.763671 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:24.763927 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:24.764356 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 11:26:24.764595 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":282,"threads_started":1}
W20260929 11:26:24.766156 24320 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33310
Sep 29 11:26:24 dist-test-slave-qk05 krb5kdc[24171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 11:26:24 dist-test-slave-qk05 krb5kdc[24171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:26:24.812366 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:24.797095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.23.153.1:33667)
0929 11:26:24.797351 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:24.797357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:24.797605 (+ 248us) server_negotiation.cc:407] Connection header received
0929 11:26:24.797720 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:24.797724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:24.797794 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:24.797872 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:24.799224 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:24.799957 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:24.801218 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:24.801689 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:24.801794 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:24.806156 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:24.806162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:24.806166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:24.806208 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:24.809739 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:24.810552 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:24.810557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:24.810559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:24.810735 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:24.811150 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:24.811153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:24.811155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:24.811431 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:24.811623 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:24.811870 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 11:26:24.812077 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 11:26:24.814006 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 24277
I20260929 11:26:24.828907 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:24.815318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.23.153.1:33667)
0929 11:26:24.815523 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:24.815528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:24.815644 (+ 116us) server_negotiation.cc:407] Connection header received
0929 11:26:24.815782 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:24.815786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:24.815845 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:24.815941 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:24.817182 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:24.817889 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:24.819288 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:24.819739 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:24.819844 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:24.822128 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:24.822133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:24.822137 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:24.822174 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:24.825914 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:24.826838 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:24.826844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:24.826847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:24.827131 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:24.827510 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:24.827513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:24.827515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:24.827799 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:24.828002 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:24.828392 (+ 390us) server_negotiation.cc:299] Negotiation successful
0929 11:26:24.828581 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:26:24.829653 24400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:24.915783 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 24277
I20260929 11:26:24.929355 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 24411
Sep 29 11:26:24 dist-test-slave-qk05 krb5kdc[24171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.2@KRBTEST.COM
I20260929 11:26:24.949251 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:24.930292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59912 (local address 127.23.153.2:42611)
0929 11:26:24.930842 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:24.930849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:24.930876 (+ 27us) server_negotiation.cc:407] Connection header received
0929 11:26:24.930937 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:24.930943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:24.931239 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:24.931381 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:24.933313 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:24.934608 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:24.936224 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:24.937304 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:24.937493 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:24.941897 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:24.941915 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:24.941932 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:24.941983 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:24.945162 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:24.946229 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:24.946241 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:24.946244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:24.946510 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:24.947407 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:24.947413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:24.947415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:24.947907 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:24.948223 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:24.948664 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 11:26:24.948904 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":265,"threads_started":1}
I20260929 11:26:24.950555 24534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:25.040083 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 24411
I20260929 11:26:25.054960 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 24545
Sep 29 11:26:25 dist-test-slave-qk05 krb5kdc[24171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.3@KRBTEST.COM
I20260929 11:26:25.072956 24690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:25.056045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.23.153.3:45965)
0929 11:26:25.056722 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:25.056730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:25.056760 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:26:25.056821 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:25.056827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:25.057048 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:25.057190 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:25.058656 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:25.059826 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:25.061223 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:25.061864 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:25.061987 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:25.066786 (+ 4799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:25.066804 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:25.066822 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:25.066864 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:25.069906 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:25.070748 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:25.070757 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:25.070759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:25.070899 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:25.071303 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:25.071307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:25.071309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:25.071690 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:25.071993 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:25.072317 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 11:26:25.072605 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":359,"threads_started":1}
I20260929 11:26:25.074332 24668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:25.159319 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 24545
I20260929 11:26:25.172284 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 24184
I20260929 11:26:25.173645 24246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:25.277336 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 24184
2026-09-29T11:26:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4080 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:26:25 dist-test-slave-qk05 krb5kdc[24695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:26:25 dist-test-slave-qk05 krb5kdc[24695](info): set up 2 sockets
Sep 29 11:26:25 dist-test-slave-qk05 krb5kdc[24695](info): commencing operation
krb5kdc: starting...
W20260929 11:26:26.286916 24164 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:26:26 dist-test-slave-qk05 krb5kdc[24695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:26:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:26:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:26:26.466032 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:38057
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:35593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:38057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:26.806775 24708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:26.807156 24708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:26.807298 24708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:26.816849 24708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:26.817034 24708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:26.817119 24708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:26.817242 24708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:26.817310 24708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:26.828917 24708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:35593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:38057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:38057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:26.830298 24708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:26.832233 24708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:26.845060 24714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:26.845106 24713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:26.845155 24716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:26.846141 24708 server_base.cc:1034] running on GCE node
I20260929 11:26:26.846971 24708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:26.848616 24708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:26.849848 24708 hybrid_clock.cc:648] HybridClock initialized: now 1790681186849786 us; error 62 us; skew 500 ppm
I20260929 11:26:26.850203 24708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:26 dist-test-slave-qk05 krb5kdc[24695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681186, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:26.853613 24708 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:26:26.855224 24708 webserver.cc:460] Webserver started at http://127.23.153.62:40781/ using document root <none> and password file <none>
I20260929 11:26:26.855958 24708 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:26.856065 24708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:26.856344 24708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:26.859572 24708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "b6aef3eb646546a19188cb4183c81ae2"
format_stamp: "Formatted at 2026-09-29 11:26:26 on dist-test-slave-qk05"
server_key: "963d8a56b615e6cbe6f6d331bc5a97d2"
server_key_iv: "14753926e4f32d2d3ded37547c49ebbc"
server_key_version: "encryptionkey@0"
I20260929 11:26:26.860399 24708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "b6aef3eb646546a19188cb4183c81ae2"
format_stamp: "Formatted at 2026-09-29 11:26:26 on dist-test-slave-qk05"
server_key: "963d8a56b615e6cbe6f6d331bc5a97d2"
server_key_iv: "14753926e4f32d2d3ded37547c49ebbc"
server_key_version: "encryptionkey@0"
I20260929 11:26:26.866662 24708 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 11:26:26.870792 24723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:26.872736 24708 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:26:26.872972 24708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "b6aef3eb646546a19188cb4183c81ae2"
format_stamp: "Formatted at 2026-09-29 11:26:26 on dist-test-slave-qk05"
server_key: "963d8a56b615e6cbe6f6d331bc5a97d2"
server_key_iv: "14753926e4f32d2d3ded37547c49ebbc"
server_key_version: "encryptionkey@0"
I20260929 11:26:26.873215 24708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:26.908064 24708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:26.912555 24708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:26.912850 24708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:26.932060 24708 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:38057
I20260929 11:26:26.932068 24774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:38057 every 8 connection(s)
I20260929 11:26:26.933604 24708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:26:26.934325 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 24708
I20260929 11:26:26.934549 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:26:26.935330 24164 external_mini_cluster.cc:1428] Setting key bc17a07c9c3fcce1ccdcf91b9670bdf8
I20260929 11:26:26.938844 24775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 11:26:26 dist-test-slave-qk05 krb5kdc[24695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:26.951164 24775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2: Bootstrap starting.
I20260929 11:26:26.954972 24775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:26.956163 24775 log.cc:824] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:26.960016 24775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2: No bootstrap required, opened a new log
I20260929 11:26:26.960668 24778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:26.937457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.23.153.62:38057)
0929 11:26:26.938419 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:26.938433 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:26.938477 (+ 44us) server_negotiation.cc:407] Connection header received
0929 11:26:26.940332 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:26.940360 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:26.940795 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:26.941201 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:26.942758 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:26.944274 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:26.945652 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:26.946420 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:26.946616 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:26.951980 (+ 5364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:26.952004 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:26.952025 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:26.952074 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:26.955640 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:26.956449 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:26.956460 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:26.956471 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:26.956728 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:26.957221 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:26.957225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:26.957227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:26.957929 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:26.958237 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:26.958703 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 11:26:26.959231 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":492,"threads_started":1}
I20260929 11:26:26.968549 24775 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aef3eb646546a19188cb4183c81ae2" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38057 } }
I20260929 11:26:26.968976 24775 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:26.969046 24775 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6aef3eb646546a19188cb4183c81ae2, State: Initialized, Role: FOLLOWER
I20260929 11:26:26.969610 24775 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aef3eb646546a19188cb4183c81ae2" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38057 } }
I20260929 11:26:26.969834 24775 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:26:26.969983 24775 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:26:26.970165 24775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:26.972538 24775 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aef3eb646546a19188cb4183c81ae2" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38057 } }
I20260929 11:26:26.973088 24775 leader_election.cc:302] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6aef3eb646546a19188cb4183c81ae2; no voters:
I20260929 11:26:26.973676 24775 leader_election.cc:288] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:26:26.973830 24780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:26.974468 24780 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2 [term 1 LEADER]: Becoming Leader. State: Replica: b6aef3eb646546a19188cb4183c81ae2, State: Running, Role: LEADER
I20260929 11:26:26.975205 24780 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aef3eb646546a19188cb4183c81ae2" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38057 } }
I20260929 11:26:26.976275 24775 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:26:26.978041 24782 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6aef3eb646546a19188cb4183c81ae2. Latest consensus state: current_term: 1 leader_uuid: "b6aef3eb646546a19188cb4183c81ae2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aef3eb646546a19188cb4183c81ae2" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38057 } } }
I20260929 11:26:26.978609 24782 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:26.978816 24781 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6aef3eb646546a19188cb4183c81ae2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aef3eb646546a19188cb4183c81ae2" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38057 } } }
I20260929 11:26:26.979126 24781 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:26.980824 24788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:26:26.986212 24788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:26:26.997242 24788 catalog_manager.cc:1347] Generated new cluster ID: 79d645e3476f47309b55f3cb059ed472
I20260929 11:26:26.997376 24788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:26:27.011096 24788 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:26:27.013077 24788 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:26:27.037772 24788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6aef3eb646546a19188cb4183c81ae2: Generated new TSK 0
I20260929 11:26:27.039132 24788 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
I20260929 11:26:27.083174 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38057
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:35593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:27.435061 24801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:27.435367 24801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:27.435494 24801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:27.443496 24801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:27.443643 24801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:27.443805 24801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:26:27.455150 24801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:35593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38057
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:27.456543 24801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:27.457950 24801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:27.472247 24807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:27.472276 24806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:27.472311 24809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:27.473127 24801 server_base.cc:1034] running on GCE node
I20260929 11:26:27.473848 24801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:27.474889 24801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:27.476145 24801 hybrid_clock.cc:648] HybridClock initialized: now 1790681187476071 us; error 64 us; skew 500 ppm
I20260929 11:26:27.476511 24801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:27 dist-test-slave-qk05 krb5kdc[24695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681187, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:27.480119 24801 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:26:27.481841 24801 webserver.cc:460] Webserver started at http://127.23.153.1:40591/ using document root <none> and password file <none>
I20260929 11:26:27.482729 24801 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:27.482877 24801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:27.483248 24801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:27.486443 24801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "8033b3ba20864875bfa07cd0d79014be"
format_stamp: "Formatted at 2026-09-29 11:26:27 on dist-test-slave-qk05"
server_key: "3548913ccf087fe3a09320e448a0fbd6"
server_key_iv: "75024aad44587de91a879b1d65f61c45"
server_key_version: "encryptionkey@0"
I20260929 11:26:27.487440 24801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "8033b3ba20864875bfa07cd0d79014be"
format_stamp: "Formatted at 2026-09-29 11:26:27 on dist-test-slave-qk05"
server_key: "3548913ccf087fe3a09320e448a0fbd6"
server_key_iv: "75024aad44587de91a879b1d65f61c45"
server_key_version: "encryptionkey@0"
I20260929 11:26:27.493454 24801 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:26:27.497467 24816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:27.499444 24801 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:26:27.499673 24801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "8033b3ba20864875bfa07cd0d79014be"
format_stamp: "Formatted at 2026-09-29 11:26:27 on dist-test-slave-qk05"
server_key: "3548913ccf087fe3a09320e448a0fbd6"
server_key_iv: "75024aad44587de91a879b1d65f61c45"
server_key_version: "encryptionkey@0"
I20260929 11:26:27.499912 24801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:27.525589 24801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:27.530357 24801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:27.530652 24801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:27.531769 24801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:26:27.533236 24801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:27.533327 24801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:27.533452 24801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:27.533504 24801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:27.565996 24801 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:34551
I20260929 11:26:27.566061 24928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:34551 every 8 connection(s)
I20260929 11:26:27.567353 24801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:26:27.571962 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 24801
I20260929 11:26:27.572196 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:26:27.572980 24164 external_mini_cluster.cc:1428] Setting key 1f62bb16e52255c98ab90ace628ad1fc
Sep 29 11:26:27 dist-test-slave-qk05 krb5kdc[24695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681187, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.2@KRBTEST.COM" created.
I20260929 11:26:27.593844 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:27.570042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:46911 (local address 127.23.153.62:38057)
0929 11:26:27.570483 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:27.570488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:27.571776 (+ 1288us) server_negotiation.cc:407] Connection header received
0929 11:26:27.574274 (+ 2498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:27.574280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:27.574375 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:27.574502 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:27.576883 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:27.577743 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:27.580217 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:27.581115 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:27.581386 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:27.586336 (+ 4950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:27.586345 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:27.586350 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:27.586409 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:27.589991 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:27.591063 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:27.591071 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:27.591074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:27.591217 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:27.591658 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:27.591662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:27.591665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:27.591953 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:27.592091 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:27.593319 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0929 11:26:27.593575 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":238,"threads_started":1}
I20260929 11:26:27.595682 24932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:27.570660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:38057 (local address 127.23.153.1:46911)
0929 11:26:27.571513 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:27.571579 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:27.573972 (+ 2393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:27.574794 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:27.574805 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:27.575718 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:27.576624 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:27.576655 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:27.577934 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:27.577940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:27.579988 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:27.580010 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:27.581759 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:27.581780 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:27.582084 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:27.582868 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:27.582899 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:27.585816 (+ 2917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:27.590239 (+ 4423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:27.590249 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:27.590267 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:27.590873 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:27.591328 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:27.591332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:27.591335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:27.591538 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:27.592097 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:27.592105 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:27.592592 (+ 487us) client_negotiation.cc:769] Sending connection context
0929 11:26:27.593158 (+ 566us) client_negotiation.cc:240] Negotiation successful
0929 11:26:27.594051 (+ 893us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":331,"threads_started":1}
I20260929 11:26:27.597658 24929 heartbeater.cc:347] Connected to a master server at 127.23.153.62:38057
I20260929 11:26:27.598039 24929 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:27.598824 24929 heartbeater.cc:511] Master 127.23.153.62:38057 requested a full tablet report, sending...
I20260929 11:26:27.601507 24740 ts_manager.cc:194] Registered new tserver with Master: 8033b3ba20864875bfa07cd0d79014be (127.23.153.1:34551)
I20260929 11:26:27.603150 24740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:46911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
I20260929 11:26:27.616261 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38057
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:35593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:27.967118 24935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:27.967458 24935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:27.967571 24935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:27.977550 24935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:27.977720 24935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:27.977874 24935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:26:27.991641 24935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:35593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38057
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:27.992988 24935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:27.994357 24935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:28.009332 24940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:28.009366 24941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:28.009312 24943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:28.010712 24935 server_base.cc:1034] running on GCE node
I20260929 11:26:28.011462 24935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:28.012585 24935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:28.013878 24935 hybrid_clock.cc:648] HybridClock initialized: now 1790681188013761 us; error 36 us; skew 500 ppm
I20260929 11:26:28.014271 24935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:28 dist-test-slave-qk05 krb5kdc[24695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681188, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:28.017774 24935 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:26:28.020270 24935 webserver.cc:460] Webserver started at http://127.23.153.2:37437/ using document root <none> and password file <none>
I20260929 11:26:28.021332 24935 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:28.021505 24935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:28.021915 24935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:28.026479 24935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "cae6d678736c4d9c932520c74740a33f"
format_stamp: "Formatted at 2026-09-29 11:26:28 on dist-test-slave-qk05"
server_key: "9e1b21061997bf6086c2f884d02b50c1"
server_key_iv: "790006f6a5784b4931e57bb119242d94"
server_key_version: "encryptionkey@0"
I20260929 11:26:28.027779 24935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "cae6d678736c4d9c932520c74740a33f"
format_stamp: "Formatted at 2026-09-29 11:26:28 on dist-test-slave-qk05"
server_key: "9e1b21061997bf6086c2f884d02b50c1"
server_key_iv: "790006f6a5784b4931e57bb119242d94"
server_key_version: "encryptionkey@0"
I20260929 11:26:28.034847 24935 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260929 11:26:28.039222 24950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:28.041167 24935 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:26:28.041396 24935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "cae6d678736c4d9c932520c74740a33f"
format_stamp: "Formatted at 2026-09-29 11:26:28 on dist-test-slave-qk05"
server_key: "9e1b21061997bf6086c2f884d02b50c1"
server_key_iv: "790006f6a5784b4931e57bb119242d94"
server_key_version: "encryptionkey@0"
I20260929 11:26:28.041607 24935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:28.061758 24935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:28.066329 24935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:28.066625 24935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:28.067739 24935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:26:28.069152 24935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:28.069235 24935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:28.069362 24935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:28.069414 24935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:28.103142 24935 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:36141
I20260929 11:26:28.103227 25062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:36141 every 8 connection(s)
I20260929 11:26:28.104714 24935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:26:28.115005 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 24935
I20260929 11:26:28.115231 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:26:28.115772 24164 external_mini_cluster.cc:1428] Setting key b4310b2c33bd954aace8d2aefa017aeb
Sep 29 11:26:28 dist-test-slave-qk05 krb5kdc[24695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681188, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:28.129582 25065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.107631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:39827 (local address 127.23.153.62:38057)
0929 11:26:28.108261 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:28.108266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:28.109283 (+ 1017us) server_negotiation.cc:407] Connection header received
0929 11:26:28.111276 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:28.111283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:28.111376 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:28.111473 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:28.113355 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.114124 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.116783 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.117356 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.117506 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.122376 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:28.122383 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:28.122386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:28.122429 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:28.125789 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:28.126792 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:28.126799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:28.126802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:28.126977 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:28.127470 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:28.127475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:28.127479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:28.127875 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:28.128030 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:28.128893 (+ 863us) server_negotiation.cc:299] Negotiation successful
0929 11:26:28.129153 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":464,"thread_start_us":364,"threads_started":1}
I20260929 11:26:28.130505 25066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.108254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:38057 (local address 127.23.153.2:39827)
0929 11:26:28.109092 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:28.109132 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:28.110898 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:28.111782 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:28.111792 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:28.112480 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:28.113134 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:28.113155 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.114308 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.114312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:28.116561 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:28.116585 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.117652 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.117661 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:28.117962 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.118722 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:28.118769 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:28.121971 (+ 3202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:28.126034 (+ 4063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:28.126043 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:28.126061 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:28.126629 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:28.127115 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:28.127119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:28.127121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:28.127352 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:28.128058 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:28.128066 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:28.128465 (+ 399us) client_negotiation.cc:769] Sending connection context
0929 11:26:28.128737 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 11:26:28.129197 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":305,"threads_started":1}
I20260929 11:26:28.132184 25063 heartbeater.cc:347] Connected to a master server at 127.23.153.62:38057
I20260929 11:26:28.132586 25063 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.153.3@KRBTEST.COM; defaulting to no policy
I20260929 11:26:28.133621 25063 heartbeater.cc:511] Master 127.23.153.62:38057 requested a full tablet report, sending...
I20260929 11:26:28.135345 24740 ts_manager.cc:194] Registered new tserver with Master: cae6d678736c4d9c932520c74740a33f (127.23.153.2:36141)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.3@KRBTEST.COM" created.
I20260929 11:26:28.136456 24740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:39827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
I20260929 11:26:28.158016 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38057
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:35593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:28.522904 25069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:28.523304 25069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:28.523422 25069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:28.531529 25069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:28.531664 25069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:28.531783 25069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:26:28.543450 25069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:35593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38057
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:28.545114 25069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:28.546630 25069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:28.561928 25075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:28.562064 25077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:28.563267 25074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:28.563613 25069 server_base.cc:1034] running on GCE node
I20260929 11:26:28.565003 25069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:28.567144 25069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:28.568464 25069 hybrid_clock.cc:648] HybridClock initialized: now 1790681188568404 us; error 75 us; skew 500 ppm
I20260929 11:26:28.568878 25069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:28 dist-test-slave-qk05 krb5kdc[24695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681188, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:28.572247 25069 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:26:28.573987 25069 webserver.cc:460] Webserver started at http://127.23.153.3:34749/ using document root <none> and password file <none>
I20260929 11:26:28.574764 25069 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:28.574903 25069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:28.575304 25069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:28.578686 25069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "9341c8b21163497093924b6c167567a4"
format_stamp: "Formatted at 2026-09-29 11:26:28 on dist-test-slave-qk05"
server_key: "674a9645e16748bc6aa74709e20d373c"
server_key_iv: "ca26d411bc7b9b93b2e0ba021dc40404"
server_key_version: "encryptionkey@0"
I20260929 11:26:28.579594 25069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "9341c8b21163497093924b6c167567a4"
format_stamp: "Formatted at 2026-09-29 11:26:28 on dist-test-slave-qk05"
server_key: "674a9645e16748bc6aa74709e20d373c"
server_key_iv: "ca26d411bc7b9b93b2e0ba021dc40404"
server_key_version: "encryptionkey@0"
I20260929 11:26:28.585616 25069 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:26:28.589829 25084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:28.591441 25069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:26:28.591626 25069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "9341c8b21163497093924b6c167567a4"
format_stamp: "Formatted at 2026-09-29 11:26:28 on dist-test-slave-qk05"
server_key: "674a9645e16748bc6aa74709e20d373c"
server_key_iv: "ca26d411bc7b9b93b2e0ba021dc40404"
server_key_version: "encryptionkey@0"
I20260929 11:26:28.591850 25069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:28.607432 24929 heartbeater.cc:503] Master 127.23.153.62:38057 was elected leader, sending a full tablet report...
I20260929 11:26:28.622151 25069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:28.626920 25069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:28.627242 25069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:28.628518 25069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:26:28.629984 25069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:28.630062 25069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:28.630174 25069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:28.630239 25069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:28.663002 25069 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:43905
I20260929 11:26:28.663106 25196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:43905 every 8 connection(s)
I20260929 11:26:28.664494 25069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:26:28.667222 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 25069
I20260929 11:26:28.667444 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:26:28.667943 24164 external_mini_cluster.cc:1428] Setting key 4d60bc6fcb4d6296408d6d23c8271d16
Sep 29 11:26:28 dist-test-slave-qk05 krb5kdc[24695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681188, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:28.688557 25199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.667367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:34687 (local address 127.23.153.62:38057)
0929 11:26:28.667903 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:28.667909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:28.669470 (+ 1561us) server_negotiation.cc:407] Connection header received
0929 11:26:28.671739 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:28.671744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:28.671837 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:28.671933 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:28.674085 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.674997 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.676751 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.677222 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.677375 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.681215 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:28.681220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:28.681224 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:28.681263 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:28.685095 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:28.685988 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:28.685994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:28.685998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:28.686148 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:28.686600 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:28.686606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:28.686609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:28.686907 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:28.687079 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:28.687899 (+ 820us) server_negotiation.cc:299] Negotiation successful
0929 11:26:28.688161 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":240,"threads_started":1}
I20260929 11:26:28.689569 25200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.668023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:38057 (local address 127.23.153.3:34687)
0929 11:26:28.669291 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:28.669342 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:28.671460 (+ 2118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:28.672242 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:28.672253 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:28.673079 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:28.673857 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:28.673888 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.675174 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.675179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:28.676589 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:28.676605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.677485 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.677490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:28.677750 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.678516 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:28.678540 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:28.680949 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:28.685288 (+ 4339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:28.685298 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:28.685314 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:28.685848 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:28.686271 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:28.686276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:28.686280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:28.686485 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:28.687066 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:28.687076 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:28.687500 (+ 424us) client_negotiation.cc:769] Sending connection context
0929 11:26:28.687772 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 11:26:28.688363 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"thread_start_us":733,"threads_started":1}
I20260929 11:26:28.691834 25197 heartbeater.cc:347] Connected to a master server at 127.23.153.62:38057
I20260929 11:26:28.692237 25197 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:28.693257 25197 heartbeater.cc:511] Master 127.23.153.62:38057 requested a full tablet report, sending...
I20260929 11:26:28.695235 24740 ts_manager.cc:194] Registered new tserver with Master: 9341c8b21163497093924b6c167567a4 (127.23.153.3:43905)
I20260929 11:26:28.696295 24740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:34687
I20260929 11:26:28.697379 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:26:28.718930 25199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.704158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54692 (local address 127.23.153.62:38057)
0929 11:26:28.704452 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:28.704458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:28.704800 (+ 342us) server_negotiation.cc:407] Connection header received
0929 11:26:28.704974 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:28.704978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:28.705050 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:28.705137 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:28.706760 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.707572 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.708960 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.709385 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.709504 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.712065 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:28.712072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:28.712075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:28.712121 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:28.715564 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:28.716723 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:28.716730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:28.716734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:28.716914 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:28.717372 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:28.717377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:28.717379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:28.717738 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:28.717991 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:28.718350 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 11:26:28.718612 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 11:26:28.726310 24740 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:26:28.733654 24740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:26:28.764319 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.750143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:36141 (local address 127.0.0.1:46754)
0929 11:26:28.751324 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:28.751374 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:28.751586 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:28.753836 (+ 2250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:28.753846 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:28.753885 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:28.755458 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:28.755474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.757754 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.757761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:28.760538 (+ 2777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:28.760570 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.763068 (+ 2498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.763077 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:28.763380 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.763453 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 11:26:28.763596 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 11:26:28.763836 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":1746,"threads_started":1}
I20260929 11:26:28.767668 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.750332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46754 (local address 127.23.153.2:36141)
0929 11:26:28.752763 (+ 2431us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:28.752772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:28.752806 (+ 34us) server_negotiation.cc:407] Connection header received
0929 11:26:28.752924 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:28.752932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:28.753270 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:28.753509 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:28.755688 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.757508 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.760903 (+ 3395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.762694 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.766576 (+ 3882us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.766885 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:28.767006 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 11:26:28.767227 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2265,"thread_start_us":2165,"threads_started":1}
I20260929 11:26:28.773677 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.750953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52844 (local address 127.23.153.1:34551)
0929 11:26:28.764199 (+ 13246us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:28.764211 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:28.764254 (+ 43us) server_negotiation.cc:407] Connection header received
0929 11:26:28.764395 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:28.764406 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:28.764789 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:28.765080 (+ 291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:28.765809 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.768009 (+ 2200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.770765 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.772411 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.772576 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.772827 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:28.773130 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 11:26:28.773334 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13005,"thread_start_us":14348,"threads_started":1}
I20260929 11:26:28.773028 24998 tablet_service.cc:1467] Processing CreateTablet for tablet 82a58d33752a49b1ba5d7cdb0da40ec8 (DEFAULT_TABLE table=test-table [id=fe15536506344ee9b9217e0326877655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:28.774210 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.750420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:34551 (local address 127.0.0.1:52844)
0929 11:26:28.752171 (+ 1751us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:28.752189 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:28.752293 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:28.765089 (+ 12796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:28.765097 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:28.765144 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:28.765585 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:28.765599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.768212 (+ 2613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.768219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:28.770515 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:28.770533 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.772698 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.772705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:28.772910 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.772947 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 11:26:28.773073 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 11:26:28.773734 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1630,"mutex_wait_us":1,"thread_start_us":719,"threads_started":1}
I20260929 11:26:28.775427 24998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82a58d33752a49b1ba5d7cdb0da40ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:28.777104 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.752957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:43905 (local address 127.0.0.1:46454)
0929 11:26:28.766197 (+ 13240us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:28.766222 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:28.766376 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:28.769011 (+ 2635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:28.769016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:28.769049 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:28.769350 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:28.769364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.771597 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.771603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:28.773414 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:28.773440 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.775873 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.775880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:28.776253 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.776306 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 11:26:28.776397 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 11:26:28.776537 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13094,"mutex_wait_us":34,"thread_start_us":15595,"threads_started":1}
I20260929 11:26:28.778673 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.753093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46454 (local address 127.23.153.3:43905)
0929 11:26:28.768278 (+ 15185us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:28.768288 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:28.768323 (+ 35us) server_negotiation.cc:407] Connection header received
0929 11:26:28.768433 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:28.768442 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:28.768745 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:28.769542 (+ 797us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:28.769836 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.771408 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.773672 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.775602 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.777733 (+ 2131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.778109 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:28.778196 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 11:26:28.778377 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14985,"thread_start_us":15700,"threads_started":1}
I20260929 11:26:28.778960 24864 tablet_service.cc:1467] Processing CreateTablet for tablet 82a58d33752a49b1ba5d7cdb0da40ec8 (DEFAULT_TABLE table=test-table [id=fe15536506344ee9b9217e0326877655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:28.780882 24864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82a58d33752a49b1ba5d7cdb0da40ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:28.789129 25132 tablet_service.cc:1467] Processing CreateTablet for tablet 82a58d33752a49b1ba5d7cdb0da40ec8 (DEFAULT_TABLE table=test-table [id=fe15536506344ee9b9217e0326877655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:28.790561 25132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82a58d33752a49b1ba5d7cdb0da40ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:28.793385 25216 tablet_bootstrap.cc:492] T 82a58d33752a49b1ba5d7cdb0da40ec8 P cae6d678736c4d9c932520c74740a33f: Bootstrap starting.
I20260929 11:26:28.797262 25217 tablet_bootstrap.cc:492] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be: Bootstrap starting.
I20260929 11:26:28.797552 25216 tablet_bootstrap.cc:654] T 82a58d33752a49b1ba5d7cdb0da40ec8 P cae6d678736c4d9c932520c74740a33f: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:28.799969 25216 log.cc:824] T 82a58d33752a49b1ba5d7cdb0da40ec8 P cae6d678736c4d9c932520c74740a33f: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:28.801019 25217 tablet_bootstrap.cc:654] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:28.802266 25217 log.cc:824] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:28.803836 25216 tablet_bootstrap.cc:492] T 82a58d33752a49b1ba5d7cdb0da40ec8 P cae6d678736c4d9c932520c74740a33f: No bootstrap required, opened a new log
I20260929 11:26:28.804163 25216 ts_tablet_manager.cc:1397] T 82a58d33752a49b1ba5d7cdb0da40ec8 P cae6d678736c4d9c932520c74740a33f: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 11:26:28.805358 25217 tablet_bootstrap.cc:492] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be: No bootstrap required, opened a new log
I20260929 11:26:28.805691 25217 ts_tablet_manager.cc:1397] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 11:26:28.807844 25220 tablet_bootstrap.cc:492] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 9341c8b21163497093924b6c167567a4: Bootstrap starting.
I20260929 11:26:28.811791 25220 tablet_bootstrap.cc:654] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 9341c8b21163497093924b6c167567a4: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:28.813274 25220 log.cc:824] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 9341c8b21163497093924b6c167567a4: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:28.814841 25216 raft_consensus.cc:348] T 82a58d33752a49b1ba5d7cdb0da40ec8 P cae6d678736c4d9c932520c74740a33f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae6d678736c4d9c932520c74740a33f" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36141 } } peers { permanent_uuid: "8033b3ba20864875bfa07cd0d79014be" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34551 } } peers { permanent_uuid: "9341c8b21163497093924b6c167567a4" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 43905 } }
I20260929 11:26:28.815320 25216 raft_consensus.cc:374] T 82a58d33752a49b1ba5d7cdb0da40ec8 P cae6d678736c4d9c932520c74740a33f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:28.815438 25216 raft_consensus.cc:729] T 82a58d33752a49b1ba5d7cdb0da40ec8 P cae6d678736c4d9c932520c74740a33f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae6d678736c4d9c932520c74740a33f, State: Initialized, Role: FOLLOWER
I20260929 11:26:28.815382 25217 raft_consensus.cc:348] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae6d678736c4d9c932520c74740a33f" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36141 } } peers { permanent_uuid: "8033b3ba20864875bfa07cd0d79014be" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34551 } } peers { permanent_uuid: "9341c8b21163497093924b6c167567a4" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 43905 } }
I20260929 11:26:28.815889 25217 raft_consensus.cc:374] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:28.816004 25217 raft_consensus.cc:729] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8033b3ba20864875bfa07cd0d79014be, State: Initialized, Role: FOLLOWER
I20260929 11:26:28.816082 25216 consensus_queue.cc:260] T 82a58d33752a49b1ba5d7cdb0da40ec8 P cae6d678736c4d9c932520c74740a33f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae6d678736c4d9c932520c74740a33f" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36141 } } peers { permanent_uuid: "8033b3ba20864875bfa07cd0d79014be" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34551 } } peers { permanent_uuid: "9341c8b21163497093924b6c167567a4" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 43905 } }
I20260929 11:26:28.816622 25217 consensus_queue.cc:260] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae6d678736c4d9c932520c74740a33f" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36141 } } peers { permanent_uuid: "8033b3ba20864875bfa07cd0d79014be" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34551 } } peers { permanent_uuid: "9341c8b21163497093924b6c167567a4" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 43905 } }
I20260929 11:26:28.817355 25063 heartbeater.cc:503] Master 127.23.153.62:38057 was elected leader, sending a full tablet report...
I20260929 11:26:28.817881 25216 ts_tablet_manager.cc:1428] T 82a58d33752a49b1ba5d7cdb0da40ec8 P cae6d678736c4d9c932520c74740a33f: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
I20260929 11:26:28.817057 25220 tablet_bootstrap.cc:492] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 9341c8b21163497093924b6c167567a4: No bootstrap required, opened a new log
I20260929 11:26:28.818396 25220 ts_tablet_manager.cc:1397] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 9341c8b21163497093924b6c167567a4: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260929 11:26:28.827677 25220 raft_consensus.cc:348] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 9341c8b21163497093924b6c167567a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae6d678736c4d9c932520c74740a33f" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36141 } } peers { permanent_uuid: "8033b3ba20864875bfa07cd0d79014be" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34551 } } peers { permanent_uuid: "9341c8b21163497093924b6c167567a4" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 43905 } }
I20260929 11:26:28.828140 25220 raft_consensus.cc:374] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 9341c8b21163497093924b6c167567a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:28.828238 25220 raft_consensus.cc:729] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 9341c8b21163497093924b6c167567a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9341c8b21163497093924b6c167567a4, State: Initialized, Role: FOLLOWER
I20260929 11:26:28.828593 25217 ts_tablet_manager.cc:1428] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be: Time spent starting tablet: real 0.023s user 0.006s sys 0.015s
I20260929 11:26:28.828984 25220 consensus_queue.cc:260] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 9341c8b21163497093924b6c167567a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae6d678736c4d9c932520c74740a33f" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36141 } } peers { permanent_uuid: "8033b3ba20864875bfa07cd0d79014be" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34551 } } peers { permanent_uuid: "9341c8b21163497093924b6c167567a4" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 43905 } }
I20260929 11:26:28.830225 25197 heartbeater.cc:503] Master 127.23.153.62:38057 was elected leader, sending a full tablet report...
I20260929 11:26:28.830981 25220 ts_tablet_manager.cc:1428] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 9341c8b21163497093924b6c167567a4: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260929 11:26:28.829912 25223 raft_consensus.cc:482] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:26:28.831506 25223 raft_consensus.cc:504] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae6d678736c4d9c932520c74740a33f" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36141 } } peers { permanent_uuid: "8033b3ba20864875bfa07cd0d79014be" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34551 } } peers { permanent_uuid: "9341c8b21163497093924b6c167567a4" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 43905 } }
I20260929 11:26:28.836030 25223 leader_election.cc:288] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cae6d678736c4d9c932520c74740a33f (127.23.153.2:36141), 9341c8b21163497093924b6c167567a4 (127.23.153.3:43905)
I20260929 11:26:28.840735 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.834319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:35471 (local address 127.23.153.2:36141)
0929 11:26:28.834583 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:28.834588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:28.834671 (+ 83us) server_negotiation.cc:407] Connection header received
0929 11:26:28.834935 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:28.834940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:28.835041 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:28.835104 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:28.835929 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.836869 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.839085 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.839858 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.840006 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.840081 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:28.840360 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 11:26:28.840489 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260929 11:26:28.840864 25225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.833849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:36141 (local address 127.23.153.1:35471)
0929 11:26:28.834580 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:28.834601 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:28.834783 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:28.835314 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:28.835318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:28.835344 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:28.835758 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:28.835777 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.837010 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.837015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:28.838899 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:28.838912 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.840056 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.840062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:28.840210 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.840234 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 11:26:28.840323 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 11:26:28.840459 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":389,"threads_started":1}
I20260929 11:26:28.841867 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82a58d33752a49b1ba5d7cdb0da40ec8" candidate_uuid: "8033b3ba20864875bfa07cd0d79014be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae6d678736c4d9c932520c74740a33f" is_pre_election: true
I20260929 11:26:28.842077 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.835313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:50993 (local address 127.23.153.3:43905)
0929 11:26:28.835492 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:28.835498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:28.836232 (+ 734us) server_negotiation.cc:407] Connection header received
0929 11:26:28.836454 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:28.836460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:28.836530 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:28.836618 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:28.837064 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.838131 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.839986 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.840746 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.840912 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.840983 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:28.841699 (+ 716us) server_negotiation.cc:299] Negotiation successful
0929 11:26:28.841841 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:26:28.842115 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.834864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:43905 (local address 127.23.153.1:50993)
0929 11:26:28.836125 (+ 1261us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:28.836146 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:28.836320 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:28.836633 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:28.836636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:28.836654 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:28.836929 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:28.836936 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.838289 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.838293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:28.839815 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:28.839829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.841073 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.841081 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:28.841541 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.841567 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 11:26:28.841651 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 11:26:28.841784 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1143,"thread_start_us":936,"threads_started":1}
I20260929 11:26:28.842411 25018 raft_consensus.cc:2454] T 82a58d33752a49b1ba5d7cdb0da40ec8 P cae6d678736c4d9c932520c74740a33f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8033b3ba20864875bfa07cd0d79014be in term 0.
I20260929 11:26:28.843120 24817 leader_election.cc:302] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8033b3ba20864875bfa07cd0d79014be, cae6d678736c4d9c932520c74740a33f; no voters:
I20260929 11:26:28.843062 25152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82a58d33752a49b1ba5d7cdb0da40ec8" candidate_uuid: "8033b3ba20864875bfa07cd0d79014be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9341c8b21163497093924b6c167567a4" is_pre_election: true
I20260929 11:26:28.843556 25152 raft_consensus.cc:2454] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 9341c8b21163497093924b6c167567a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8033b3ba20864875bfa07cd0d79014be in term 0.
I20260929 11:26:28.843710 25223 raft_consensus.cc:2781] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:26:28.843871 25223 raft_consensus.cc:482] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:26:28.843978 25223 raft_consensus.cc:3037] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:28.846231 25223 raft_consensus.cc:504] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae6d678736c4d9c932520c74740a33f" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36141 } } peers { permanent_uuid: "8033b3ba20864875bfa07cd0d79014be" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34551 } } peers { permanent_uuid: "9341c8b21163497093924b6c167567a4" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 43905 } }
I20260929 11:26:28.846812 25223 leader_election.cc:288] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be [CANDIDATE]: Term 1 election: Requested vote from peers cae6d678736c4d9c932520c74740a33f (127.23.153.2:36141), 9341c8b21163497093924b6c167567a4 (127.23.153.3:43905)
I20260929 11:26:28.847397 25018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82a58d33752a49b1ba5d7cdb0da40ec8" candidate_uuid: "8033b3ba20864875bfa07cd0d79014be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae6d678736c4d9c932520c74740a33f"
I20260929 11:26:28.847437 25152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82a58d33752a49b1ba5d7cdb0da40ec8" candidate_uuid: "8033b3ba20864875bfa07cd0d79014be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9341c8b21163497093924b6c167567a4"
I20260929 11:26:28.847594 25018 raft_consensus.cc:3037] T 82a58d33752a49b1ba5d7cdb0da40ec8 P cae6d678736c4d9c932520c74740a33f [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:28.847633 25152 raft_consensus.cc:3037] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 9341c8b21163497093924b6c167567a4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:28.850198 25152 raft_consensus.cc:2454] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 9341c8b21163497093924b6c167567a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8033b3ba20864875bfa07cd0d79014be in term 1.
I20260929 11:26:28.850610 24817 leader_election.cc:302] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8033b3ba20864875bfa07cd0d79014be, 9341c8b21163497093924b6c167567a4; no voters:
I20260929 11:26:28.850711 25018 raft_consensus.cc:2454] T 82a58d33752a49b1ba5d7cdb0da40ec8 P cae6d678736c4d9c932520c74740a33f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8033b3ba20864875bfa07cd0d79014be in term 1.
I20260929 11:26:28.850936 25223 raft_consensus.cc:2781] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:28.851437 25223 raft_consensus.cc:686] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be [term 1 LEADER]: Becoming Leader. State: Replica: 8033b3ba20864875bfa07cd0d79014be, State: Running, Role: LEADER
I20260929 11:26:28.851931 25223 consensus_queue.cc:237] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae6d678736c4d9c932520c74740a33f" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36141 } } peers { permanent_uuid: "8033b3ba20864875bfa07cd0d79014be" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34551 } } peers { permanent_uuid: "9341c8b21163497093924b6c167567a4" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 43905 } }
W20260929 11:26:28.856902 25064 tablet.cc:2369] T 82a58d33752a49b1ba5d7cdb0da40ec8 P cae6d678736c4d9c932520c74740a33f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:26:28.856423 24740 catalog_manager.cc:5495] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be reported cstate change: term changed from 0 to 1, leader changed from <none> to 8033b3ba20864875bfa07cd0d79014be (127.23.153.1). New cstate: current_term: 1 leader_uuid: "8033b3ba20864875bfa07cd0d79014be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae6d678736c4d9c932520c74740a33f" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8033b3ba20864875bfa07cd0d79014be" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9341c8b21163497093924b6c167567a4" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 43905 } health_report { overall_health: UNKNOWN } } }
Sep 29 11:26:28 dist-test-slave-qk05 krb5kdc[24695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:26:28.887676 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.872044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52860 (local address 127.23.153.1:34551)
0929 11:26:28.872264 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:28.872271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:28.872752 (+ 481us) server_negotiation.cc:407] Connection header received
0929 11:26:28.872923 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:28.872927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:28.873003 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:28.873115 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:28.874577 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.875500 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.876888 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.877335 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.877504 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.881801 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:28.881820 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:28.881838 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:28.881881 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:28.884752 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:28.885536 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:28.885544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:28.885547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:28.885706 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:28.886089 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:28.886092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:28.886095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:28.886452 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:28.886736 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:28.887070 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 11:26:28.887357 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 11:26:28 dist-test-slave-qk05 krb5kdc[24695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681188, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260929 11:26:28.916220 25198 tablet.cc:2369] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 9341c8b21163497093924b6c167567a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 11:26:28 dist-test-slave-qk05 krb5kdc[24695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681188, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:26:28.935105 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.919181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.23.153.1:34551)
0929 11:26:28.919346 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:28.919352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:28.919802 (+ 450us) server_negotiation.cc:407] Connection header received
0929 11:26:28.919987 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:28.919991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:28.920066 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:28.920163 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:28.921549 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.922407 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.924336 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.924833 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.924998 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.929426 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:28.929432 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:28.929436 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:28.929480 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:28.932148 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:28.932881 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:28.932886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:28.932888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:28.933034 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:28.933530 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:28.933536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:28.933539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:28.933877 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:28.934211 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:28.934481 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 11:26:28.934718 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 11:26:28 dist-test-slave-qk05 krb5kdc[24695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 11:26:28 dist-test-slave-qk05 krb5kdc[24695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:26:28.982519 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.967445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.23.153.1:34551)
0929 11:26:28.967639 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:28.967645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:28.968123 (+ 478us) server_negotiation.cc:407] Connection header received
0929 11:26:28.968313 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:28.968319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:28.968403 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:28.968489 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:28.969934 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.970727 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.972036 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.972481 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.972611 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.976811 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:28.976817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:28.976821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:28.976867 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:28.979542 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:28.980408 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:28.980414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:28.980418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:28.980592 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:28.980925 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:28.980931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:28.980935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:28.981334 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:28.981611 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:28.981874 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 11:26:28.982120 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 11:26:28.984920 24844 tablet_service.cc:2908] Rejecting scan request for tablet 82a58d33752a49b1ba5d7cdb0da40ec8: Uninitialized: clean time has not yet been initialized
I20260929 11:26:28.988322 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 24801
I20260929 11:26:29.002462 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:28.989845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52886 (local address 127.23.153.1:34551)
0929 11:26:28.990103 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:28.990110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:28.990352 (+ 242us) server_negotiation.cc:407] Connection header received
0929 11:26:28.990496 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:28.990500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:28.990584 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:28.990673 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:28.991918 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.992656 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.994036 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:28.994455 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:28.994566 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:28.997068 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:28.997075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:28.997080 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:28.997133 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:28.999790 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:29.000632 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:29.000637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:29.000639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:29.000813 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:29.001134 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:29.001138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:29.001140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:29.001432 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:29.001632 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:29.001961 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 11:26:29.002199 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:26:29.003234 24924 generic_service.cc:196] Checking for leaks (request via RPC)
W20260929 11:26:29.095431 24930 tablet.cc:2369] T 82a58d33752a49b1ba5d7cdb0da40ec8 P 8033b3ba20864875bfa07cd0d79014be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:26:29.096128 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 24801
I20260929 11:26:29.111243 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 24935
Sep 29 11:26:29 dist-test-slave-qk05 krb5kdc[24695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.2@KRBTEST.COM
I20260929 11:26:29.132628 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:29.112515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.23.153.2:36141)
0929 11:26:29.112710 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:29.112727 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:29.113000 (+ 273us) server_negotiation.cc:407] Connection header received
0929 11:26:29.113255 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:29.113261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:29.113404 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:29.113522 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:29.115441 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:29.116275 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:29.117879 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:29.118319 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:29.118425 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:29.126134 (+ 7709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:29.126153 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:29.126171 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:29.126215 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:29.129456 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:29.130284 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:29.130293 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:29.130295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:29.130447 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:29.130909 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:29.130914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:29.130916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:29.131401 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:29.131652 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:29.132122 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 11:26:29.132342 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 11:26:29.133430 25058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:29.223956 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 24935
I20260929 11:26:29.238032 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 25069
Sep 29 11:26:29 dist-test-slave-qk05 krb5kdc[24695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.3@KRBTEST.COM
I20260929 11:26:29.254911 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:29.238884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46460 (local address 127.23.153.3:43905)
0929 11:26:29.239113 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:29.239119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:29.239140 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:26:29.239258 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:29.239264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:29.239331 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:29.239416 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:29.240852 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:29.241603 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:29.242952 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:29.243490 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:29.243625 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:29.248898 (+ 5273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:29.248917 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:29.248936 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:29.248981 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:29.251923 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:29.252775 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:29.252783 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:29.252786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:29.252939 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:29.253335 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:29.253340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:29.253342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:29.253727 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:29.254024 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:29.254348 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 11:26:29.254575 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 11:26:29.255590 25192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:29.343636 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 25069
I20260929 11:26:29.357172 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 24708
I20260929 11:26:29.358395 24770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:29.468837 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 24708
2026-09-29T11:26:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4193 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:26:29 dist-test-slave-qk05 krb5kdc[25246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:26:29 dist-test-slave-qk05 krb5kdc[25246](info): set up 2 sockets
Sep 29 11:26:29 dist-test-slave-qk05 krb5kdc[25246](info): commencing operation
krb5kdc: starting...
W20260929 11:26:30.486953 24164 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:26:30 dist-test-slave-qk05 krb5kdc[25246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:26:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:26:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:26:30.678809 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:45917
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:38341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:45917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:31.027112 25259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:31.027789 25259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:31.027951 25259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:31.037058 25259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:31.037204 25259 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 11:26:31.037258 25259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:31.037358 25259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:31.037410 25259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:31.037463 25259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:31.048324 25259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:38341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:45917
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:45917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:31.049753 25259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:31.051218 25259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:31.063346 25264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:31.063354 25267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:31.063297 25265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:31.064079 25259 server_base.cc:1034] running on GCE node
I20260929 11:26:31.064932 25259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:31.066530 25259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:31.067804 25259 hybrid_clock.cc:648] HybridClock initialized: now 1790681191067729 us; error 47 us; skew 500 ppm
I20260929 11:26:31.068214 25259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:31 dist-test-slave-qk05 krb5kdc[25246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681191, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:31.071619 25259 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:26:31.073318 25259 webserver.cc:460] Webserver started at http://127.23.153.62:37873/ using document root <none> and password file <none>
I20260929 11:26:31.074092 25259 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:31.074193 25259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:31.074494 25259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:31.078015 25259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "4b0ee9ea79cc4f788b552e6cecfb7124"
format_stamp: "Formatted at 2026-09-29 11:26:31 on dist-test-slave-qk05"
server_key: "2496d1db05bffd7cff5625e41c7edeea"
server_key_iv: "143377eb9a40b1853a9bfb0ff7b17252"
server_key_version: "encryptionkey@0"
I20260929 11:26:31.078876 25259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "4b0ee9ea79cc4f788b552e6cecfb7124"
format_stamp: "Formatted at 2026-09-29 11:26:31 on dist-test-slave-qk05"
server_key: "2496d1db05bffd7cff5625e41c7edeea"
server_key_iv: "143377eb9a40b1853a9bfb0ff7b17252"
server_key_version: "encryptionkey@0"
I20260929 11:26:31.084735 25259 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 11:26:31.088842 25274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:31.090428 25259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:26:31.090622 25259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "4b0ee9ea79cc4f788b552e6cecfb7124"
format_stamp: "Formatted at 2026-09-29 11:26:31 on dist-test-slave-qk05"
server_key: "2496d1db05bffd7cff5625e41c7edeea"
server_key_iv: "143377eb9a40b1853a9bfb0ff7b17252"
server_key_version: "encryptionkey@0"
I20260929 11:26:31.090858 25259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:31.105067 25259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:31.109812 25259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:31.110091 25259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:31.131708 25259 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:45917
I20260929 11:26:31.131719 25335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:45917 every 8 connection(s)
I20260929 11:26:31.133155 25259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:26:31.137610 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 25259
I20260929 11:26:31.137822 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:26:31.137941 25336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:31.138510 24164 external_mini_cluster.cc:1428] Setting key 0ebcfbf12f95d756d57c0fce3654f4c0
I20260929 11:26:31.154145 25336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124: Bootstrap starting.
Sep 29 11:26:31 dist-test-slave-qk05 krb5kdc[25246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:31.157634 25336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:31.159164 25336 log.cc:824] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:31.162698 25336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124: No bootstrap required, opened a new log
I20260929 11:26:31.166775 25339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:31.140827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36700 (local address 127.23.153.62:45917)
0929 11:26:31.141765 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:31.141783 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:31.141845 (+ 62us) server_negotiation.cc:407] Connection header received
0929 11:26:31.145435 (+ 3590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:31.145481 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:31.146391 (+ 910us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:31.146983 (+ 592us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:31.148380 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:31.150385 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:31.151909 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:31.153087 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:31.153294 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:31.157473 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:31.157486 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:31.157513 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:31.157578 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:31.161492 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:31.162985 (+ 1493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:31.162993 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:31.163000 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:31.163293 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:31.163737 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:31.163741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:31.163743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:31.164345 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:31.164643 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:31.165161 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 11:26:31.165617 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":396,"threads_started":1}
I20260929 11:26:31.172348 25336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0ee9ea79cc4f788b552e6cecfb7124" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45917 } }
I20260929 11:26:31.172778 25336 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:31.172932 25336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0ee9ea79cc4f788b552e6cecfb7124, State: Initialized, Role: FOLLOWER
I20260929 11:26:31.173607 25336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0ee9ea79cc4f788b552e6cecfb7124" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45917 } }
I20260929 11:26:31.173900 25336 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:26:31.174045 25336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:26:31.174193 25336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:31.176962 25336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0ee9ea79cc4f788b552e6cecfb7124" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45917 } }
I20260929 11:26:31.177511 25336 leader_election.cc:302] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b0ee9ea79cc4f788b552e6cecfb7124; no voters:
I20260929 11:26:31.178066 25336 leader_election.cc:288] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:26:31.178203 25341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:31.179319 25341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124 [term 1 LEADER]: Becoming Leader. State: Replica: 4b0ee9ea79cc4f788b552e6cecfb7124, State: Running, Role: LEADER
I20260929 11:26:31.179917 25341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0ee9ea79cc4f788b552e6cecfb7124" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45917 } }
I20260929 11:26:31.180280 25336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:26:31.182360 25343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b0ee9ea79cc4f788b552e6cecfb7124. Latest consensus state: current_term: 1 leader_uuid: "4b0ee9ea79cc4f788b552e6cecfb7124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0ee9ea79cc4f788b552e6cecfb7124" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45917 } } }
I20260929 11:26:31.182691 25343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:31.183266 25342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b0ee9ea79cc4f788b552e6cecfb7124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b0ee9ea79cc4f788b552e6cecfb7124" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45917 } } }
I20260929 11:26:31.183539 25342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:31.183562 25348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:26:31.189788 25348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:26:31.199645 25348 catalog_manager.cc:1347] Generated new cluster ID: 71db81311aaf463f977c675fc056efcd
I20260929 11:26:31.199787 25348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:26:31.211241 25348 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:26:31.213310 25348 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:26:31.233407 25348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124: Generated new TSK 0
I20260929 11:26:31.234527 25348 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
I20260929 11:26:31.298938 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:45917
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:38341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:31.697341 25362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:31.697669 25362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:31.697768 25362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:31.708046 25362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:31.708216 25362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:31.708359 25362 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:26:31.708490 25362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:26:31.720085 25362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:38341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.153.62:45917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:31.721480 25362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:31.722930 25362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:31.739708 25367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:31.739603 25370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:31.739629 25368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:31.740691 25362 server_base.cc:1034] running on GCE node
I20260929 11:26:31.741412 25362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:31.742435 25362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:31.743724 25362 hybrid_clock.cc:648] HybridClock initialized: now 1790681191743654 us; error 41 us; skew 500 ppm
I20260929 11:26:31.744050 25362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:31 dist-test-slave-qk05 krb5kdc[25246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681191, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:31.747413 25362 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:26:31.749014 25362 webserver.cc:460] Webserver started at http://127.23.153.1:34999/ using document root <none> and password file <none>
I20260929 11:26:31.749756 25362 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:31.749848 25362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:31.750144 25362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:31.753355 25362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "fe6ba3ca201a4969b10f93862746aa7f"
format_stamp: "Formatted at 2026-09-29 11:26:31 on dist-test-slave-qk05"
server_key: "f78322afec8db37e149f7006d893d105"
server_key_iv: "735b68e8ebcde821bcdc21e78f414e0f"
server_key_version: "encryptionkey@0"
I20260929 11:26:31.754199 25362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "fe6ba3ca201a4969b10f93862746aa7f"
format_stamp: "Formatted at 2026-09-29 11:26:31 on dist-test-slave-qk05"
server_key: "f78322afec8db37e149f7006d893d105"
server_key_iv: "735b68e8ebcde821bcdc21e78f414e0f"
server_key_version: "encryptionkey@0"
I20260929 11:26:31.761631 25362 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260929 11:26:31.766237 25377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:31.768364 25362 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 11:26:31.768657 25362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "fe6ba3ca201a4969b10f93862746aa7f"
format_stamp: "Formatted at 2026-09-29 11:26:31 on dist-test-slave-qk05"
server_key: "f78322afec8db37e149f7006d893d105"
server_key_iv: "735b68e8ebcde821bcdc21e78f414e0f"
server_key_version: "encryptionkey@0"
I20260929 11:26:31.768913 25362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:31.784251 25362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:31.788983 25362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:31.789242 25362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:31.792135 25362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:31.792244 25362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:31.792389 25362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:31.792480 25362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:26:31 dist-test-slave-qk05 krb5kdc[25246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681191, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:31.818552 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:31.792693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:42747 (local address 127.23.153.62:45917)
0929 11:26:31.793422 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:31.793429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:31.794782 (+ 1353us) server_negotiation.cc:407] Connection header received
0929 11:26:31.797280 (+ 2498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:31.797291 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:31.797414 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:31.797783 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:31.800302 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:31.801722 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:31.804523 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:31.805198 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:31.805373 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:31.810776 (+ 5403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:31.810783 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:31.810787 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:31.810838 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:31.813977 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:31.815385 (+ 1408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:31.815390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:31.815393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:31.815542 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:31.816223 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:31.816226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:31.816228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:31.816516 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:31.816639 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:31.818074 (+ 1435us) server_negotiation.cc:299] Negotiation successful
0929 11:26:31.818277 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":471,"threads_started":1}
I20260929 11:26:31.820596 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:31.793426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:45917 (local address 127.23.153.1:42747)
0929 11:26:31.794447 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:31.794539 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:31.796734 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:31.798083 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:31.798094 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:31.799087 (+ 993us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:31.799952 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:31.799979 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:31.801945 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:31.801952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:31.804284 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:31.804313 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:31.805541 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:31.805550 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:31.805815 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:31.806896 (+ 1081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:31.806932 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:31.810380 (+ 3448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:31.814252 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:31.814267 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:31.814291 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:31.815202 (+ 911us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:31.815708 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:31.815716 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:31.815721 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:31.816071 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:31.816909 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:31.816921 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:31.817523 (+ 602us) client_negotiation.cc:769] Sending connection context
0929 11:26:31.817961 (+ 438us) client_negotiation.cc:240] Negotiation successful
0929 11:26:31.818637 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":376,"threads_started":1}
I20260929 11:26:31.842331 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:31.828396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:56383 (local address 127.23.153.62:45917)
0929 11:26:31.828587 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:31.828594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:31.829316 (+ 722us) server_negotiation.cc:407] Connection header received
0929 11:26:31.829506 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:31.829518 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:31.829595 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:31.829676 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:31.830649 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:31.831539 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:31.833334 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:31.833776 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:31.833894 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:31.836504 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:31.836520 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:31.836524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:31.836572 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:31.839734 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:31.840431 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:31.840437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:31.840441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:31.840582 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:31.840931 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:31.840937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:31.840941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:31.841220 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:31.841372 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:31.841761 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 11:26:31.841982 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:26:31.842469 25468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:31.828522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:45917 (local address 127.23.153.1:56383)
0929 11:26:31.829191 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:31.829211 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:31.829379 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:31.829720 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:31.829725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:31.830202 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:31.830491 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:31.830501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:31.831691 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:31.831697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:31.833188 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:31.833202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:31.833978 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:31.833984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:31.834183 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:31.834918 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:31.834943 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:31.836200 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:31.839878 (+ 3678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:31.839885 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:31.839890 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:31.840308 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:31.840694 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:31.840701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:31.840705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:31.840830 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:31.841335 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:31.841341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:31.841526 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 11:26:31.841649 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 11:26:31.842035 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":355,"threads_started":1}
I20260929 11:26:31.844166 25362 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:45357
I20260929 11:26:31.844208 25497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:45357 every 8 connection(s)
I20260929 11:26:31.845643 25362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:26:31.847904 25498 heartbeater.cc:347] Connected to a master server at 127.23.153.62:45917
I20260929 11:26:31.848142 25498 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:31.848805 25498 heartbeater.cc:511] Master 127.23.153.62:45917 requested a full tablet report, sending...
I20260929 11:26:31.850987 25291 ts_manager.cc:194] Registered new tserver with Master: fe6ba3ca201a4969b10f93862746aa7f (127.23.153.1:45357)
I20260929 11:26:31.851929 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 25362
I20260929 11:26:31.852120 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:26:31.852624 25291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:42747
I20260929 11:26:31.852728 24164 external_mini_cluster.cc:1428] Setting key dda90885c6a799543eb55a2cf2b9fb2f
WARNING: no policy specified for kudu/127.23.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
I20260929 11:26:31.890535 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:45917
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:38341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:32.241863 25502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:32.242178 25502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:32.242273 25502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:32.249853 25502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:32.249996 25502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:32.250084 25502 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:26:32.250166 25502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:26:32.260926 25502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:38341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.153.62:45917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:32.262283 25502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:32.263778 25502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:32.278635 25510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:32.278856 25508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:32.278899 25507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:32.280066 25502 server_base.cc:1034] running on GCE node
I20260929 11:26:32.280767 25502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:32.281817 25502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:32.283092 25502 hybrid_clock.cc:648] HybridClock initialized: now 1790681192282992 us; error 49 us; skew 500 ppm
I20260929 11:26:32.283398 25502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:32 dist-test-slave-qk05 krb5kdc[25246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681192, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:32.287230 25502 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:26:32.289158 25502 webserver.cc:460] Webserver started at http://127.23.153.2:36881/ using document root <none> and password file <none>
I20260929 11:26:32.289978 25502 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:32.290093 25502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:32.290439 25502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:32.293692 25502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "b22b2717539348d984d5014d1909cecf"
format_stamp: "Formatted at 2026-09-29 11:26:32 on dist-test-slave-qk05"
server_key: "2e627ff231a818f64535ba84911ff63b"
server_key_iv: "0f5802573d7e7b42c71c5432f945f212"
server_key_version: "encryptionkey@0"
I20260929 11:26:32.294602 25502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "b22b2717539348d984d5014d1909cecf"
format_stamp: "Formatted at 2026-09-29 11:26:32 on dist-test-slave-qk05"
server_key: "2e627ff231a818f64535ba84911ff63b"
server_key_iv: "0f5802573d7e7b42c71c5432f945f212"
server_key_version: "encryptionkey@0"
I20260929 11:26:32.300609 25502 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 11:26:32.304718 25517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:32.306332 25502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:26:32.306596 25502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "b22b2717539348d984d5014d1909cecf"
format_stamp: "Formatted at 2026-09-29 11:26:32 on dist-test-slave-qk05"
server_key: "2e627ff231a818f64535ba84911ff63b"
server_key_iv: "0f5802573d7e7b42c71c5432f945f212"
server_key_version: "encryptionkey@0"
I20260929 11:26:32.306895 25502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:32.321444 25502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:32.326292 25502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:32.326588 25502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:32.329723 25502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:32.329855 25502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:32.330062 25502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:32.330180 25502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 29 11:26:32 dist-test-slave-qk05 krb5kdc[25246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681192, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:32.357643 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:32.330188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:35203 (local address 127.23.153.62:45917)
0929 11:26:32.330371 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:32.330376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:32.332223 (+ 1847us) server_negotiation.cc:407] Connection header received
0929 11:26:32.334377 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:32.334383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:32.334478 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:32.334690 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:32.337215 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:32.338347 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:32.341719 (+ 3372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:32.342396 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:32.342567 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:32.348558 (+ 5991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:32.348564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:32.348568 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:32.348608 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:32.352090 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:32.353374 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:32.353381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:32.353385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:32.353540 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:32.354470 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:32.354474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:32.354476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:32.354795 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:32.354964 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:32.356748 (+ 1784us) server_negotiation.cc:299] Negotiation successful
0929 11:26:32.357041 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 11:26:32.359195 25526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:32.331153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:45917 (local address 127.23.153.2:35203)
0929 11:26:32.332032 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:32.332075 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:32.334010 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:32.335060 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:32.335075 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:32.335918 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:32.336915 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:32.336967 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:32.338751 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:32.338757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:32.341493 (+ 2736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:32.341519 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:32.342762 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:32.342772 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:32.343230 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:32.344237 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:32.344267 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:32.348177 (+ 3910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:32.352307 (+ 4130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:32.352319 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:32.352343 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:32.353197 (+ 854us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:32.353877 (+ 680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:32.353883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:32.353887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:32.354268 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:32.355100 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:32.355110 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:32.355719 (+ 609us) client_negotiation.cc:769] Sending connection context
0929 11:26:32.356062 (+ 343us) client_negotiation.cc:240] Negotiation successful
0929 11:26:32.356781 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":394,"threads_started":1}
I20260929 11:26:32.371387 25502 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:39003
I20260929 11:26:32.371404 25636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:39003 every 8 connection(s)
I20260929 11:26:32.372884 25502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:26:32.375373 25637 heartbeater.cc:347] Connected to a master server at 127.23.153.62:45917
I20260929 11:26:32.375641 25637 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:32.376456 25637 heartbeater.cc:511] Master 127.23.153.62:45917 requested a full tablet report, sending...
I20260929 11:26:32.378274 25291 ts_manager.cc:194] Registered new tserver with Master: b22b2717539348d984d5014d1909cecf (127.23.153.2:39003)
I20260929 11:26:32.379251 25291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:35203
I20260929 11:26:32.379604 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 25502
I20260929 11:26:32.379801 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:26:32.380329 24164 external_mini_cluster.cc:1428] Setting key 044855d81b8232dc6f1f90aebb35dc11
I20260929 11:26:32.382886 25629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:32.367427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:45917 (local address 127.23.153.2:59175)
0929 11:26:32.368216 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:32.368239 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:32.368456 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:32.368835 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:32.368840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:32.369528 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:32.369856 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:32.369867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:32.371081 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:32.371086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:32.372550 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:32.372585 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:32.373710 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:32.373719 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:32.373921 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:32.375281 (+ 1360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:32.375307 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:32.376212 (+ 905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:32.379801 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:32.379807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:32.379811 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:32.380387 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:32.380938 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:32.380942 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:32.380946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:32.381104 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:32.381792 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:32.381797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:32.381987 (+ 190us) client_negotiation.cc:769] Sending connection context
0929 11:26:32.382111 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 11:26:32.382432 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":403,"threads_started":1}
I20260929 11:26:32.383836 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:32.367333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:59175 (local address 127.23.153.62:45917)
0929 11:26:32.367552 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:32.367558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:32.368374 (+ 816us) server_negotiation.cc:407] Connection header received
0929 11:26:32.368594 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:32.368597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:32.368667 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:32.368770 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:32.370027 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:32.370901 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:32.372800 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:32.373469 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:32.373642 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:32.376431 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:32.376436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:32.376440 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:32.376484 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:32.379642 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:32.380567 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:32.380573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:32.380576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:32.380796 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:32.381239 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:32.381244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:32.381247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:32.381673 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:32.381862 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:32.383278 (+ 1416us) server_negotiation.cc:299] Negotiation successful
0929 11:26:32.383540 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
WARNING: no policy specified for kudu/127.23.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
I20260929 11:26:32.423070 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:45917
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:38341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:32.773108 25641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:32.773468 25641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:32.773598 25641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:32.781359 25641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:32.781500 25641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:32.781618 25641 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:26:32.781723 25641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:26:32.792092 25641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:38341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.153.62:45917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:32.793488 25641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:32.795147 25641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:32.809191 25646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:32.809175 25649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:32.809175 25647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:32.810461 25641 server_base.cc:1034] running on GCE node
I20260929 11:26:32.811407 25641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:32.812531 25641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:32.813818 25641 hybrid_clock.cc:648] HybridClock initialized: now 1790681192813736 us; error 48 us; skew 500 ppm
I20260929 11:26:32.814186 25641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:32 dist-test-slave-qk05 krb5kdc[25246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681192, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:32.817749 25641 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:26:32.819662 25641 webserver.cc:460] Webserver started at http://127.23.153.3:42503/ using document root <none> and password file <none>
I20260929 11:26:32.820418 25641 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:32.820533 25641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:32.820860 25641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:32.824154 25641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "59496649db32403cb1065957b5768435"
format_stamp: "Formatted at 2026-09-29 11:26:32 on dist-test-slave-qk05"
server_key: "de4a38071caf81c348b222c44ea3d98a"
server_key_iv: "9a8c7b4c0015171de10528b79de83a5b"
server_key_version: "encryptionkey@0"
I20260929 11:26:32.825057 25641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "59496649db32403cb1065957b5768435"
format_stamp: "Formatted at 2026-09-29 11:26:32 on dist-test-slave-qk05"
server_key: "de4a38071caf81c348b222c44ea3d98a"
server_key_iv: "9a8c7b4c0015171de10528b79de83a5b"
server_key_version: "encryptionkey@0"
I20260929 11:26:32.831564 25641 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 11:26:32.835804 25656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:32.837414 25641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:26:32.837637 25641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "59496649db32403cb1065957b5768435"
format_stamp: "Formatted at 2026-09-29 11:26:32 on dist-test-slave-qk05"
server_key: "de4a38071caf81c348b222c44ea3d98a"
server_key_iv: "9a8c7b4c0015171de10528b79de83a5b"
server_key_version: "encryptionkey@0"
I20260929 11:26:32.837872 25641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:32.856069 25498 heartbeater.cc:503] Master 127.23.153.62:45917 was elected leader, sending a full tablet report...
I20260929 11:26:32.863164 25641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:32.868317 25641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:32.868609 25641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:32.871502 25641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:32.871603 25641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:32.871770 25641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:32.871836 25641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:26:32 dist-test-slave-qk05 krb5kdc[25246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681192, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:32.896406 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:32.872383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:58657 (local address 127.23.153.62:45917)
0929 11:26:32.872605 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:32.872610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:32.874764 (+ 2154us) server_negotiation.cc:407] Connection header received
0929 11:26:32.877109 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:32.877117 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:32.877227 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:32.877354 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:32.880158 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:32.881216 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:32.884075 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:32.884639 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:32.884794 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:32.889965 (+ 5171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:32.889970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:32.889974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:32.890024 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:32.892713 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:32.893599 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:32.893604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:32.893606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:32.893749 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:32.894208 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:32.894212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:32.894214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:32.894505 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:32.894636 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:32.895786 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0929 11:26:32.896038 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260929 11:26:32.897957 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:32.873626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:45917 (local address 127.23.153.3:58657)
0929 11:26:32.874568 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:32.874616 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:32.876636 (+ 2020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:32.877784 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:32.877800 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:32.878786 (+ 986us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:32.879880 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:32.879917 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:32.881493 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:32.881500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:32.883869 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:32.883892 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:32.884971 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:32.884987 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:32.885435 (+ 448us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:32.886366 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:32.886420 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:32.889663 (+ 3243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:32.892909 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:32.892922 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:32.892940 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:32.893443 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:32.893868 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:32.893874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:32.893878 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:32.894083 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:32.894711 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:32.894721 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:32.895294 (+ 573us) client_negotiation.cc:769] Sending connection context
0929 11:26:32.895626 (+ 332us) client_negotiation.cc:240] Negotiation successful
0929 11:26:32.896318 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":367,"threads_started":1}
I20260929 11:26:32.920571 25641 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:37411
I20260929 11:26:32.920580 25775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:37411 every 8 connection(s)
I20260929 11:26:32.922331 25641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:26:32.924721 25776 heartbeater.cc:347] Connected to a master server at 127.23.153.62:45917
I20260929 11:26:32.925004 25776 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:32.925931 25776 heartbeater.cc:511] Master 127.23.153.62:45917 requested a full tablet report, sending...
I20260929 11:26:32.926366 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:32.909334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:45917 (local address 127.23.153.3:33815)
0929 11:26:32.910065 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:32.910084 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:32.910256 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:32.910607 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:32.910612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:32.911353 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:32.911680 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:32.911691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:32.913173 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:32.913180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:32.915217 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:32.915246 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:32.916156 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:32.916164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:32.916370 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:32.917227 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:32.917261 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:32.918431 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:32.922616 (+ 4185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:32.922621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:32.922624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:32.923285 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:32.923789 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:32.923794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:32.923797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:32.924051 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:32.925074 (+ 1023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:32.925078 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:32.925296 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 11:26:32.925452 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 11:26:32.925841 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":583,"threads_started":1}
I20260929 11:26:32.926419 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:32.909852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:33815 (local address 127.23.153.62:45917)
0929 11:26:32.910038 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:32.910044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:32.910170 (+ 126us) server_negotiation.cc:407] Connection header received
0929 11:26:32.910378 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:32.910384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:32.910469 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:32.910553 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:32.911863 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:32.912931 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:32.915453 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:32.915912 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:32.916032 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:32.918709 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:32.918717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:32.918722 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:32.918767 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:32.922424 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:32.923446 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:32.923454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:32.923458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:32.923649 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:32.924220 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:32.924225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:32.924228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:32.924821 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:32.924982 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:32.925735 (+ 753us) server_negotiation.cc:299] Negotiation successful
0929 11:26:32.926147 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 11:26:32.927639 25290 ts_manager.cc:194] Registered new tserver with Master: 59496649db32403cb1065957b5768435 (127.23.153.3:37411)
I20260929 11:26:32.928508 25290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:58657
I20260929 11:26:32.932677 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 25641
I20260929 11:26:32.932848 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:26:32.933259 24164 external_mini_cluster.cc:1428] Setting key f460122d3685abe9629808ee6489f3a0
I20260929 11:26:32.936936 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 11:26:32 dist-test-slave-qk05 krb5kdc[25246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681192, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 11:26:32 dist-test-slave-qk05 krb5kdc[25246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681192, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:32.986898 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:32.968770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36710 (local address 127.23.153.62:45917)
0929 11:26:32.969057 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:32.969063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:32.969738 (+ 675us) server_negotiation.cc:407] Connection header received
0929 11:26:32.969965 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:32.969970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:32.970060 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:32.970195 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:32.971838 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:32.973059 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:32.974668 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:32.975388 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:32.975564 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:32.980467 (+ 4903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:32.980474 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:32.980478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:32.980525 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:32.983363 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:32.984288 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:32.984293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:32.984295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:32.984649 (+ 354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:32.985125 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:32.985129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:32.985132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:32.985529 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:32.985887 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:32.986278 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 11:26:32.986587 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260929 11:26:32.992672 25290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:26:32.999925 25290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:26:33.025238 25794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:33.016157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.23.153.3:37411)
0929 11:26:33.017287 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:33.017297 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:33.018017 (+ 720us) server_negotiation.cc:407] Connection header received
0929 11:26:33.018277 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:33.018284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:33.018523 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:33.018717 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:33.019315 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.020681 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.023136 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.024167 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.024290 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:33.024516 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:33.024752 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 11:26:33.024933 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959,"thread_start_us":806,"threads_started":1}
I20260929 11:26:33.025262 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:33.015701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:37411 (local address 127.0.0.1:58458)
0929 11:26:33.017887 (+ 2186us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:33.017914 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:33.018052 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:33.018718 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:33.018723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:33.018747 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:33.019099 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:33.019111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.020857 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.020865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:33.022957 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:33.022967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.024372 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.024378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:33.024586 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:33.024609 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 11:26:33.024686 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 11:26:33.024824 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2055,"mutex_wait_us":28,"thread_start_us":324,"threads_started":1}
I20260929 11:26:33.025300 25789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:33.013841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:45357 (local address 127.0.0.1:43584)
0929 11:26:33.015279 (+ 1438us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:33.015331 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:33.015504 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:33.018302 (+ 2798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:33.018307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:33.018332 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:33.018641 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:33.018651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.020238 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.020244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:33.022595 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:33.022610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.024543 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.024550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:33.024703 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:33.024735 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 11:26:33.024801 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 11:26:33.024924 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"mutex_wait_us":40,"thread_start_us":974,"threads_started":1}
I20260929 11:26:33.026273 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:33.014018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43584 (local address 127.23.153.1:45357)
0929 11:26:33.017696 (+ 3678us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:33.017705 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:33.017733 (+ 28us) server_negotiation.cc:407] Connection header received
0929 11:26:33.017826 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:33.017834 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:33.018078 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:33.018312 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:33.018828 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.020104 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.022775 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.024294 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.025547 (+ 1253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:33.025750 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:33.025818 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 11:26:33.025993 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3540,"thread_start_us":3417,"threads_started":1}
I20260929 11:26:33.027230 25793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:33.014394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:39003 (local address 127.0.0.1:41382)
0929 11:26:33.016278 (+ 1884us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:33.016302 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:33.016419 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:33.017472 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:33.017479 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:33.017508 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:33.017987 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:33.018002 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.019750 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.019756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:33.022048 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:33.022060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.023974 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.023981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:33.024226 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:33.024282 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 11:26:33.024409 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 11:26:33.024633 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1762,"mutex_wait_us":72,"thread_start_us":2240,"threads_started":1}
I20260929 11:26:33.027604 25792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:33.014933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41382 (local address 127.23.153.2:39003)
0929 11:26:33.016577 (+ 1644us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:33.016588 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:33.016618 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:26:33.016720 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:33.016729 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:33.017019 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:33.017220 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:33.018179 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.019609 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.022214 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.023470 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.023611 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:33.023848 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:33.026052 (+ 2204us) server_negotiation.cc:299] Negotiation successful
0929 11:26:33.026227 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1475,"thread_start_us":1353,"threads_started":1}
I20260929 11:26:33.029539 25706 tablet_service.cc:1467] Processing CreateTablet for tablet 30220f982e6b473b86f2b0ac44c30169 (DEFAULT_TABLE table=test-table [id=042d4d6dc53c41b98ea169073e42f7c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:33.029623 25428 tablet_service.cc:1467] Processing CreateTablet for tablet 30220f982e6b473b86f2b0ac44c30169 (DEFAULT_TABLE table=test-table [id=042d4d6dc53c41b98ea169073e42f7c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:33.031139 25428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30220f982e6b473b86f2b0ac44c30169. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:33.031211 25706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30220f982e6b473b86f2b0ac44c30169. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:33.031515 25567 tablet_service.cc:1467] Processing CreateTablet for tablet 30220f982e6b473b86f2b0ac44c30169 (DEFAULT_TABLE table=test-table [id=042d4d6dc53c41b98ea169073e42f7c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:33.033000 25567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30220f982e6b473b86f2b0ac44c30169. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:33.043454 25795 tablet_bootstrap.cc:492] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435: Bootstrap starting.
I20260929 11:26:33.045917 25796 tablet_bootstrap.cc:492] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f: Bootstrap starting.
I20260929 11:26:33.047290 25795 tablet_bootstrap.cc:654] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:33.048691 25795 log.cc:824] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:33.049854 25796 tablet_bootstrap.cc:654] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:33.051568 25796 log.cc:824] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:33.052091 25797 tablet_bootstrap.cc:492] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf: Bootstrap starting.
I20260929 11:26:33.054513 25795 tablet_bootstrap.cc:492] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435: No bootstrap required, opened a new log
I20260929 11:26:33.054873 25795 ts_tablet_manager.cc:1397] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20260929 11:26:33.057780 25796 tablet_bootstrap.cc:492] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f: No bootstrap required, opened a new log
I20260929 11:26:33.057770 25797 tablet_bootstrap.cc:654] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:33.058104 25796 ts_tablet_manager.cc:1397] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 11:26:33.059263 25797 log.cc:824] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:33.063865 25797 tablet_bootstrap.cc:492] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf: No bootstrap required, opened a new log
I20260929 11:26:33.064172 25797 ts_tablet_manager.cc:1397] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260929 11:26:33.065328 25795 raft_consensus.cc:348] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } }
I20260929 11:26:33.065793 25795 raft_consensus.cc:374] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:33.065886 25795 raft_consensus.cc:729] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59496649db32403cb1065957b5768435, State: Initialized, Role: FOLLOWER
I20260929 11:26:33.066668 25795 consensus_queue.cc:260] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } }
I20260929 11:26:33.067942 25776 heartbeater.cc:503] Master 127.23.153.62:45917 was elected leader, sending a full tablet report...
I20260929 11:26:33.068674 25795 ts_tablet_manager.cc:1428] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260929 11:26:33.068570 25796 raft_consensus.cc:348] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } }
I20260929 11:26:33.069032 25796 raft_consensus.cc:374] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:33.069130 25796 raft_consensus.cc:729] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6ba3ca201a4969b10f93862746aa7f, State: Initialized, Role: FOLLOWER
I20260929 11:26:33.069895 25796 consensus_queue.cc:260] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } }
I20260929 11:26:33.072254 25796 ts_tablet_manager.cc:1428] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260929 11:26:33.074308 25797 raft_consensus.cc:348] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } }
I20260929 11:26:33.074724 25797 raft_consensus.cc:374] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:33.074832 25797 raft_consensus.cc:729] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22b2717539348d984d5014d1909cecf, State: Initialized, Role: FOLLOWER
I20260929 11:26:33.075489 25797 consensus_queue.cc:260] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } }
I20260929 11:26:33.076714 25637 heartbeater.cc:503] Master 127.23.153.62:45917 was elected leader, sending a full tablet report...
I20260929 11:26:33.077580 25797 ts_tablet_manager.cc:1428] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
W20260929 11:26:33.098373 25499 tablet.cc:2369] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:26:33.125950 25638 tablet.cc:2369] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:26:33.174921 25777 tablet.cc:2369] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:26:33.316264 25801 raft_consensus.cc:482] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:26:33.316603 25801 raft_consensus.cc:504] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } }
I20260929 11:26:33.318259 25801 leader_election.cc:288] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe6ba3ca201a4969b10f93862746aa7f (127.23.153.1:45357), b22b2717539348d984d5014d1909cecf (127.23.153.2:39003)
I20260929 11:26:33.326035 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:33.318674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:45357 (local address 127.23.153.3:37089)
0929 11:26:33.318889 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:33.318907 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:33.319065 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:33.319501 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:33.319506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:33.319550 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:33.319790 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:33.319803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.321040 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.321043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:33.322910 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:33.322932 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.325023 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.325029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:33.325314 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:33.325355 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 11:26:33.325495 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 11:26:33.325651 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":5}
I20260929 11:26:33.326077 25804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:33.319177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:39003 (local address 127.23.153.3:55805)
0929 11:26:33.319828 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:33.319844 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:33.319953 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:33.320253 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:33.320256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:33.320274 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:33.320544 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:33.320552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.321600 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.321605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:33.323525 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:33.323539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.325251 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.325257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:33.325470 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:33.325499 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 11:26:33.325584 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 11:26:33.325694 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":76,"thread_start_us":336,"threads_started":1}
I20260929 11:26:33.326164 25792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:33.318772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:55805 (local address 127.23.153.2:39003)
0929 11:26:33.318968 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:33.318974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:33.319928 (+ 954us) server_negotiation.cc:407] Connection header received
0929 11:26:33.320071 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:33.320076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:33.320147 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:33.320245 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:33.320676 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.321461 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.323744 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.324992 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.325167 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:33.325238 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:33.325653 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 11:26:33.325794 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:26:33.326742 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:33.318663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:37089 (local address 127.23.153.1:45357)
0929 11:26:33.318882 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:33.318888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:33.318978 (+ 90us) server_negotiation.cc:407] Connection header received
0929 11:26:33.319257 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:33.319263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:33.319340 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:33.319467 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:33.319963 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.320893 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.323295 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.324736 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.325044 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:33.325145 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:33.325568 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 11:26:33.325734 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:26:33.327690 25448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30220f982e6b473b86f2b0ac44c30169" candidate_uuid: "59496649db32403cb1065957b5768435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6ba3ca201a4969b10f93862746aa7f" is_pre_election: true
I20260929 11:26:33.327739 25587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30220f982e6b473b86f2b0ac44c30169" candidate_uuid: "59496649db32403cb1065957b5768435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22b2717539348d984d5014d1909cecf" is_pre_election: true
I20260929 11:26:33.328236 25448 raft_consensus.cc:2454] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59496649db32403cb1065957b5768435 in term 0.
I20260929 11:26:33.328467 25587 raft_consensus.cc:2454] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59496649db32403cb1065957b5768435 in term 0.
I20260929 11:26:33.329090 25660 leader_election.cc:302] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59496649db32403cb1065957b5768435, fe6ba3ca201a4969b10f93862746aa7f; no voters:
I20260929 11:26:33.329509 25801 raft_consensus.cc:2781] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:26:33.329668 25801 raft_consensus.cc:482] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:26:33.329769 25801 raft_consensus.cc:3037] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:33.332080 25801 raft_consensus.cc:504] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } }
I20260929 11:26:33.332778 25801 leader_election.cc:288] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [CANDIDATE]: Term 1 election: Requested vote from peers fe6ba3ca201a4969b10f93862746aa7f (127.23.153.1:45357), b22b2717539348d984d5014d1909cecf (127.23.153.2:39003)
I20260929 11:26:33.333353 25448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30220f982e6b473b86f2b0ac44c30169" candidate_uuid: "59496649db32403cb1065957b5768435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6ba3ca201a4969b10f93862746aa7f"
I20260929 11:26:33.333406 25587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30220f982e6b473b86f2b0ac44c30169" candidate_uuid: "59496649db32403cb1065957b5768435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22b2717539348d984d5014d1909cecf"
I20260929 11:26:33.333654 25587 raft_consensus.cc:3037] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:33.333664 25448 raft_consensus.cc:3037] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:33.336098 25587 raft_consensus.cc:2454] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59496649db32403cb1065957b5768435 in term 1.
I20260929 11:26:33.336145 25448 raft_consensus.cc:2454] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59496649db32403cb1065957b5768435 in term 1.
I20260929 11:26:33.336640 25658 leader_election.cc:302] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59496649db32403cb1065957b5768435, b22b2717539348d984d5014d1909cecf; no voters:
I20260929 11:26:33.336979 25801 raft_consensus.cc:2781] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:33.337360 25801 raft_consensus.cc:686] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [term 1 LEADER]: Becoming Leader. State: Replica: 59496649db32403cb1065957b5768435, State: Running, Role: LEADER
I20260929 11:26:33.337829 25801 consensus_queue.cc:237] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } }
I20260929 11:26:33.342612 25291 catalog_manager.cc:5495] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59496649db32403cb1065957b5768435 (127.23.153.3). New cstate: current_term: 1 leader_uuid: "59496649db32403cb1065957b5768435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260929 11:26:33.407269 25794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:33.401308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58468 (local address 127.23.153.3:37411)
0929 11:26:33.401545 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:33.401552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:33.401672 (+ 120us) server_negotiation.cc:407] Connection header received
0929 11:26:33.402028 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:33.402033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:33.402115 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:33.402236 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:33.402745 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.403542 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.404931 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.405347 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.405493 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:33.406029 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:33.406601 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:33.406745 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:33.406874 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 11:26:33.406976 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 11:26:33.415905 25448 raft_consensus.cc:1261] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f [term 1 FOLLOWER]: Refusing update from remote peer 59496649db32403cb1065957b5768435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:33.415907 25587 raft_consensus.cc:1261] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf [term 1 FOLLOWER]: Refusing update from remote peer 59496649db32403cb1065957b5768435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:33.416936 25801 consensus_queue.cc:1035] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:33.417265 25805 consensus_queue.cc:1035] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:33.430214 25808 mvcc.cc:205] Tried to move back new op lower bound from 7334630168221134848 to 7334630167916060672. Current Snapshot: MvccSnapshot[applied={T|T < 7334630168221134848}]
I20260929 11:26:33.453467 25291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:26:33.453771 25291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:26:33.457770 25291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124: Sending DeleteTablet for 3 replicas of tablet 30220f982e6b473b86f2b0ac44c30169
I20260929 11:26:33.459173 25567 tablet_service.cc:1514] Processing DeleteTablet for tablet 30220f982e6b473b86f2b0ac44c30169 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:26:33 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:41382
I20260929 11:26:33.459201 25428 tablet_service.cc:1514] Processing DeleteTablet for tablet 30220f982e6b473b86f2b0ac44c30169 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:26:33 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:43584
I20260929 11:26:33.459285 25706 tablet_service.cc:1514] Processing DeleteTablet for tablet 30220f982e6b473b86f2b0ac44c30169 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:26:33 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:58458
I20260929 11:26:33.462091 25291 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:26:33.462937 25291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:26:33.469833 25820 tablet_replica.cc:330] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435: stopping tablet replica
I20260929 11:26:33.470760 25820 raft_consensus.cc:2229] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:26:33.472048 25820 raft_consensus.cc:2258] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:26:33.473239 25818 tablet_replica.cc:330] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f: stopping tablet replica
I20260929 11:26:33.475229 25818 raft_consensus.cc:2229] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:26:33.476260 25706 tablet_service.cc:1467] Processing CreateTablet for tablet 124551c5dfaa4ae58a1367da9461b714 (DEFAULT_TABLE table=test-table [id=aeadf017d1544b118c9939c4c06bfc27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:33.476387 25566 tablet_service.cc:1467] Processing CreateTablet for tablet 124551c5dfaa4ae58a1367da9461b714 (DEFAULT_TABLE table=test-table [id=aeadf017d1544b118c9939c4c06bfc27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:33.476966 25706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124551c5dfaa4ae58a1367da9461b714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:33.477067 25566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124551c5dfaa4ae58a1367da9461b714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:33.486025 25797 tablet_bootstrap.cc:492] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf: Bootstrap starting.
I20260929 11:26:33.476267 25427 tablet_service.cc:1467] Processing CreateTablet for tablet 124551c5dfaa4ae58a1367da9461b714 (DEFAULT_TABLE table=test-table [id=aeadf017d1544b118c9939c4c06bfc27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:33.487190 25427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124551c5dfaa4ae58a1367da9461b714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:33.486796 25818 raft_consensus.cc:2258] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:26:33.487504 25819 tablet_replica.cc:330] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf: stopping tablet replica
I20260929 11:26:33.487668 25820 ts_tablet_manager.cc:1905] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:26:33.487915 25797 tablet_bootstrap.cc:654] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:33.487983 25819 raft_consensus.cc:2229] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:26:33.488488 25819 raft_consensus.cc:2258] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:26:33.490895 25795 tablet_bootstrap.cc:492] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435: Bootstrap starting.
I20260929 11:26:33.492678 25795 tablet_bootstrap.cc:654] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:33.493476 25819 ts_tablet_manager.cc:1905] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:26:33.497345 25796 tablet_bootstrap.cc:492] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f: Bootstrap starting.
I20260929 11:26:33.499308 25818 ts_tablet_manager.cc:1905] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:26:33.499967 25796 tablet_bootstrap.cc:654] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:33.500793 25797 tablet_bootstrap.cc:492] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf: No bootstrap required, opened a new log
I20260929 11:26:33.500989 25797 ts_tablet_manager.cc:1397] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260929 11:26:33.506733 25819 ts_tablet_manager.cc:1918] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:26:33.506974 25819 log.cc:1196] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/30220f982e6b473b86f2b0ac44c30169
I20260929 11:26:33.507634 25819 ts_tablet_manager.cc:1939] T 30220f982e6b473b86f2b0ac44c30169 P b22b2717539348d984d5014d1909cecf: Deleting consensus metadata
I20260929 11:26:33.509714 25276 catalog_manager.cc:4841] TS b22b2717539348d984d5014d1909cecf (127.23.153.2:39003): tablet 30220f982e6b473b86f2b0ac44c30169 (table test-table [id=042d4d6dc53c41b98ea169073e42f7c1]) successfully deleted
I20260929 11:26:33.510203 25797 raft_consensus.cc:348] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } }
I20260929 11:26:33.510452 25797 raft_consensus.cc:374] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:33.510545 25797 raft_consensus.cc:729] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22b2717539348d984d5014d1909cecf, State: Initialized, Role: FOLLOWER
I20260929 11:26:33.506922 25820 ts_tablet_manager.cc:1918] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:26:33.511348 25820 log.cc:1196] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/30220f982e6b473b86f2b0ac44c30169
I20260929 11:26:33.511729 25795 tablet_bootstrap.cc:492] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435: No bootstrap required, opened a new log
I20260929 11:26:33.511896 25795 ts_tablet_manager.cc:1397] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.004s
I20260929 11:26:33.511930 25796 tablet_bootstrap.cc:492] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f: No bootstrap required, opened a new log
I20260929 11:26:33.512079 25796 ts_tablet_manager.cc:1397] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20260929 11:26:33.512426 25820 ts_tablet_manager.cc:1939] T 30220f982e6b473b86f2b0ac44c30169 P 59496649db32403cb1065957b5768435: Deleting consensus metadata
I20260929 11:26:33.513850 25795 raft_consensus.cc:348] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } }
I20260929 11:26:33.514536 25795 raft_consensus.cc:374] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:33.514477 25796 raft_consensus.cc:348] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } }
I20260929 11:26:33.514729 25796 raft_consensus.cc:374] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:33.514823 25796 raft_consensus.cc:729] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6ba3ca201a4969b10f93862746aa7f, State: Initialized, Role: FOLLOWER
I20260929 11:26:33.514930 25795 raft_consensus.cc:729] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59496649db32403cb1065957b5768435, State: Initialized, Role: FOLLOWER
I20260929 11:26:33.515169 25796 consensus_queue.cc:260] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } }
I20260929 11:26:33.515612 25795 consensus_queue.cc:260] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } }
I20260929 11:26:33.516225 25278 catalog_manager.cc:4841] TS 59496649db32403cb1065957b5768435 (127.23.153.3:37411): tablet 30220f982e6b473b86f2b0ac44c30169 (table test-table [id=042d4d6dc53c41b98ea169073e42f7c1]) successfully deleted
I20260929 11:26:33.516768 25795 ts_tablet_manager.cc:1428] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 11:26:33.520272 25796 ts_tablet_manager.cc:1428] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20260929 11:26:33.521104 25818 ts_tablet_manager.cc:1918] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:26:33.521427 25818 log.cc:1196] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/30220f982e6b473b86f2b0ac44c30169
I20260929 11:26:33.522120 25818 ts_tablet_manager.cc:1939] T 30220f982e6b473b86f2b0ac44c30169 P fe6ba3ca201a4969b10f93862746aa7f: Deleting consensus metadata
I20260929 11:26:33.510951 25797 consensus_queue.cc:260] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } }
I20260929 11:26:33.524124 25797 ts_tablet_manager.cc:1428] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
I20260929 11:26:33.524377 25278 catalog_manager.cc:4841] TS fe6ba3ca201a4969b10f93862746aa7f (127.23.153.1:45357): tablet 30220f982e6b473b86f2b0ac44c30169 (table test-table [id=042d4d6dc53c41b98ea169073e42f7c1]) successfully deleted
I20260929 11:26:33.718824 25803 raft_consensus.cc:482] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:26:33.719126 25803 raft_consensus.cc:504] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } }
I20260929 11:26:33.721138 25803 leader_election.cc:288] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe6ba3ca201a4969b10f93862746aa7f (127.23.153.1:45357), 59496649db32403cb1065957b5768435 (127.23.153.3:37411)
I20260929 11:26:33.729333 25790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:33.721548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:51977 (local address 127.23.153.1:45357)
0929 11:26:33.721760 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:33.721768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:33.722333 (+ 565us) server_negotiation.cc:407] Connection header received
0929 11:26:33.722562 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:33.722570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:33.722659 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:33.722731 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:33.723852 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.725087 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.727325 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.728265 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.728459 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:33.728543 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:33.728888 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 11:26:33.729040 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:26:33.729408 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:33.721576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:45357 (local address 127.23.153.2:51977)
0929 11:26:33.722207 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:33.722227 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:33.722376 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:33.723050 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:33.723057 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:33.723089 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:33.723674 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:33.723690 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.725242 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.725247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:33.727133 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:33.727151 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.728503 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.728509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:33.728679 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:33.728717 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 11:26:33.728821 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 11:26:33.728975 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":123,"thread_start_us":357,"threads_started":1}
I20260929 11:26:33.730476 25448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124551c5dfaa4ae58a1367da9461b714" candidate_uuid: "b22b2717539348d984d5014d1909cecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6ba3ca201a4969b10f93862746aa7f" is_pre_election: true
I20260929 11:26:33.730707 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:33.722425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:37411 (local address 127.23.153.2:33491)
0929 11:26:33.723389 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:33.723412 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:33.723555 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:33.723907 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:33.723911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:33.723934 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:33.724318 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:33.724328 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.725530 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.725612 (+ 82us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:33.728042 (+ 2430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:33.728061 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.729944 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.729949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:33.730078 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:33.730103 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 11:26:33.730188 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 11:26:33.730332 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":6,"thread_start_us":572,"threads_started":1}
I20260929 11:26:33.730775 25448 raft_consensus.cc:2454] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22b2717539348d984d5014d1909cecf in term 0.
I20260929 11:26:33.731688 25521 leader_election.cc:302] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22b2717539348d984d5014d1909cecf, fe6ba3ca201a4969b10f93862746aa7f; no voters:
I20260929 11:26:33.731796 25794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:33.722603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:33491 (local address 127.23.153.3:37411)
0929 11:26:33.722784 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:33.722790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:33.723592 (+ 802us) server_negotiation.cc:407] Connection header received
0929 11:26:33.723677 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:33.723682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:33.723801 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:33.723887 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:33.724481 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.725278 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.728275 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.729449 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.729609 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:33.729681 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:33.731276 (+ 1595us) server_negotiation.cc:299] Negotiation successful
0929 11:26:33.731504 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:26:33.732190 25803 raft_consensus.cc:2781] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:26:33.732327 25803 raft_consensus.cc:482] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:26:33.732483 25803 raft_consensus.cc:3037] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:33.732780 25727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124551c5dfaa4ae58a1367da9461b714" candidate_uuid: "b22b2717539348d984d5014d1909cecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59496649db32403cb1065957b5768435" is_pre_election: true
I20260929 11:26:33.733229 25727 raft_consensus.cc:2454] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b22b2717539348d984d5014d1909cecf in term 0.
I20260929 11:26:33.735781 25803 raft_consensus.cc:504] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } }
I20260929 11:26:33.736630 25803 leader_election.cc:288] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [CANDIDATE]: Term 1 election: Requested vote from peers fe6ba3ca201a4969b10f93862746aa7f (127.23.153.1:45357), 59496649db32403cb1065957b5768435 (127.23.153.3:37411)
I20260929 11:26:33.737344 25727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124551c5dfaa4ae58a1367da9461b714" candidate_uuid: "b22b2717539348d984d5014d1909cecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59496649db32403cb1065957b5768435"
I20260929 11:26:33.737612 25727 raft_consensus.cc:3037] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:33.737553 25448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124551c5dfaa4ae58a1367da9461b714" candidate_uuid: "b22b2717539348d984d5014d1909cecf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6ba3ca201a4969b10f93862746aa7f"
I20260929 11:26:33.737896 25448 raft_consensus.cc:3037] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:33.740971 25448 raft_consensus.cc:2454] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22b2717539348d984d5014d1909cecf in term 1.
I20260929 11:26:33.741173 25727 raft_consensus.cc:2454] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b22b2717539348d984d5014d1909cecf in term 1.
I20260929 11:26:33.741593 25521 leader_election.cc:302] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b22b2717539348d984d5014d1909cecf, fe6ba3ca201a4969b10f93862746aa7f; no voters:
I20260929 11:26:33.741956 25803 raft_consensus.cc:2781] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:33.752830 25803 raft_consensus.cc:686] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [term 1 LEADER]: Becoming Leader. State: Replica: b22b2717539348d984d5014d1909cecf, State: Running, Role: LEADER
I20260929 11:26:33.753274 25803 consensus_queue.cc:237] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } }
I20260929 11:26:33.757687 25290 catalog_manager.cc:5495] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf reported cstate change: term changed from 0 to 1, leader changed from <none> to b22b2717539348d984d5014d1909cecf (127.23.153.2). New cstate: current_term: 1 leader_uuid: "b22b2717539348d984d5014d1909cecf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } health_report { overall_health: UNKNOWN } } }
Sep 29 11:26:33 dist-test-slave-qk05 krb5kdc[25246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681191, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:33.848556 25833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:33.834084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:45917 (local address 127.0.0.1:36712)
0929 11:26:33.834757 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:33.834783 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:33.834954 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:33.835484 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:33.835489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:33.836155 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:33.836371 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:33.836380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.837469 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.837473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:33.838480 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:33.838493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.839227 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.839233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:33.839407 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:33.840101 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:33.840119 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:33.842451 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:33.845830 (+ 3379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:33.845858 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:33.845868 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:33.846310 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:33.846671 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:33.846676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:33.846680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:33.846829 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:33.847420 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:33.847430 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:33.847599 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 11:26:33.847750 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 11:26:33.848092 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":1,"thread_start_us":355,"threads_started":1}
I20260929 11:26:33.848615 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:33.834097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36712 (local address 127.23.153.62:45917)
0929 11:26:33.835117 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:33.835121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:33.835138 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:26:33.835177 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:33.835180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:33.835232 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:33.835318 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:33.836547 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.837329 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.838624 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:33.839071 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:33.839181 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:33.842671 (+ 3490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:33.842675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:33.842679 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:33.842709 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:33.845656 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:33.846439 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:33.846444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:33.846446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:33.846566 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:33.846950 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:33.846954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:33.846958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:33.847281 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:33.847427 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:33.847845 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 11:26:33.848219 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":180,"server-negotiator.queue_time_us":870,"thread_start_us":732,"threads_started":1}
I20260929 11:26:33.854521 25290 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:36712:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 11:26:33.855659 25290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:26:33.862558 25427 tablet_service.cc:1467] Processing CreateTablet for tablet d50056b2f1bb4058aee120243cb5538d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d8f9980ad5204373938ffdb33143ca95]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:26:33.863201 25427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d50056b2f1bb4058aee120243cb5538d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:33.863123 25567 tablet_service.cc:1467] Processing CreateTablet for tablet d50056b2f1bb4058aee120243cb5538d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d8f9980ad5204373938ffdb33143ca95]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:26:33.863626 25567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d50056b2f1bb4058aee120243cb5538d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:33.867589 25706 tablet_service.cc:1467] Processing CreateTablet for tablet d50056b2f1bb4058aee120243cb5538d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d8f9980ad5204373938ffdb33143ca95]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:26:33.868098 25706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d50056b2f1bb4058aee120243cb5538d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:33.871251 25796 tablet_bootstrap.cc:492] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f: Bootstrap starting.
I20260929 11:26:33.874081 25796 tablet_bootstrap.cc:654] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:33.878473 25797 tablet_bootstrap.cc:492] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf: Bootstrap starting.
I20260929 11:26:33.879319 25795 tablet_bootstrap.cc:492] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435: Bootstrap starting.
I20260929 11:26:33.880345 25797 tablet_bootstrap.cc:654] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:33.881119 25795 tablet_bootstrap.cc:654] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:33.881470 25796 tablet_bootstrap.cc:492] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f: No bootstrap required, opened a new log
I20260929 11:26:33.881676 25796 ts_tablet_manager.cc:1397] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20260929 11:26:33.882565 25796 raft_consensus.cc:348] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } }
I20260929 11:26:33.882880 25796 raft_consensus.cc:374] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:33.882967 25796 raft_consensus.cc:729] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6ba3ca201a4969b10f93862746aa7f, State: Initialized, Role: FOLLOWER
I20260929 11:26:33.883318 25796 consensus_queue.cc:260] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } }
I20260929 11:26:33.884246 25796 ts_tablet_manager.cc:1428] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:26:33.884102 25802 tablet_replica.cc:439] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } }
I20260929 11:26:33.884648 25802 tablet_replica.cc:442] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:26:33.886691 25797 tablet_bootstrap.cc:492] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf: No bootstrap required, opened a new log
I20260929 11:26:33.886885 25797 ts_tablet_manager.cc:1397] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 11:26:33.887296 25795 tablet_bootstrap.cc:492] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435: No bootstrap required, opened a new log
I20260929 11:26:33.887491 25795 ts_tablet_manager.cc:1397] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20260929 11:26:33.887753 25797 raft_consensus.cc:348] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } }
I20260929 11:26:33.888005 25797 raft_consensus.cc:374] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:33.888090 25797 raft_consensus.cc:729] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22b2717539348d984d5014d1909cecf, State: Initialized, Role: FOLLOWER
I20260929 11:26:33.888374 25795 raft_consensus.cc:348] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } }
I20260929 11:26:33.888365 25797 consensus_queue.cc:260] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } }
I20260929 11:26:33.888621 25795 raft_consensus.cc:374] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:33.888713 25795 raft_consensus.cc:729] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59496649db32403cb1065957b5768435, State: Initialized, Role: FOLLOWER
I20260929 11:26:33.888926 25797 ts_tablet_manager.cc:1428] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:26:33.888978 25795 consensus_queue.cc:260] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } }
I20260929 11:26:33.889406 25803 tablet_replica.cc:439] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } }
I20260929 11:26:33.889406 25813 tablet_replica.cc:439] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } }
I20260929 11:26:33.889699 25803 tablet_replica.cc:442] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:26:33.889724 25813 tablet_replica.cc:442] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:26:33.890245 25795 ts_tablet_manager.cc:1428] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 11:26:34.033648 25813 raft_consensus.cc:482] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:26:34.033854 25813 raft_consensus.cc:504] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } }
I20260929 11:26:34.034562 25813 leader_election.cc:288] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe6ba3ca201a4969b10f93862746aa7f (127.23.153.1:45357), b22b2717539348d984d5014d1909cecf (127.23.153.2:39003)
I20260929 11:26:34.035203 25448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d50056b2f1bb4058aee120243cb5538d" candidate_uuid: "59496649db32403cb1065957b5768435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6ba3ca201a4969b10f93862746aa7f" is_pre_election: true
I20260929 11:26:34.035219 25587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d50056b2f1bb4058aee120243cb5538d" candidate_uuid: "59496649db32403cb1065957b5768435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22b2717539348d984d5014d1909cecf" is_pre_election: true
I20260929 11:26:34.035480 25448 raft_consensus.cc:2454] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59496649db32403cb1065957b5768435 in term 0.
I20260929 11:26:34.035641 25587 raft_consensus.cc:2454] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59496649db32403cb1065957b5768435 in term 0.
I20260929 11:26:34.035928 25660 leader_election.cc:302] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59496649db32403cb1065957b5768435, fe6ba3ca201a4969b10f93862746aa7f; no voters:
I20260929 11:26:34.036212 25813 raft_consensus.cc:2781] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:26:34.036338 25813 raft_consensus.cc:482] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:26:34.036526 25813 raft_consensus.cc:3037] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:34.038570 25813 raft_consensus.cc:504] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } }
I20260929 11:26:34.039230 25813 leader_election.cc:288] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [CANDIDATE]: Term 1 election: Requested vote from peers fe6ba3ca201a4969b10f93862746aa7f (127.23.153.1:45357), b22b2717539348d984d5014d1909cecf (127.23.153.2:39003)
I20260929 11:26:34.039758 25448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d50056b2f1bb4058aee120243cb5538d" candidate_uuid: "59496649db32403cb1065957b5768435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6ba3ca201a4969b10f93862746aa7f"
I20260929 11:26:34.039786 25587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d50056b2f1bb4058aee120243cb5538d" candidate_uuid: "59496649db32403cb1065957b5768435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22b2717539348d984d5014d1909cecf"
I20260929 11:26:34.040020 25587 raft_consensus.cc:3037] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:34.040024 25448 raft_consensus.cc:3037] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:34.042961 25448 raft_consensus.cc:2454] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59496649db32403cb1065957b5768435 in term 1.
I20260929 11:26:34.042968 25587 raft_consensus.cc:2454] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59496649db32403cb1065957b5768435 in term 1.
I20260929 11:26:34.043531 25660 leader_election.cc:302] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59496649db32403cb1065957b5768435, fe6ba3ca201a4969b10f93862746aa7f; no voters:
I20260929 11:26:34.043887 25813 raft_consensus.cc:2781] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:34.044272 25813 raft_consensus.cc:686] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [term 1 LEADER]: Becoming Leader. State: Replica: 59496649db32403cb1065957b5768435, State: Running, Role: LEADER
I20260929 11:26:34.044550 25813 consensus_queue.cc:237] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } }
I20260929 11:26:34.046444 25838 tablet_replica.cc:439] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435: TxnStatusTablet state changed. Reason: New leader 59496649db32403cb1065957b5768435. Latest consensus state: current_term: 1 leader_uuid: "59496649db32403cb1065957b5768435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } }
I20260929 11:26:34.046823 25838 tablet_replica.cc:442] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:26:34.047435 25840 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 11:26:34.047636 25840 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 11:26:34.047978 25290 catalog_manager.cc:5495] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59496649db32403cb1065957b5768435 (127.23.153.3). New cstate: current_term: 1 leader_uuid: "59496649db32403cb1065957b5768435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } health_report { overall_health: HEALTHY } } }
I20260929 11:26:34.097404 25794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:34.090528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58470 (local address 127.23.153.3:37411)
0929 11:26:34.090783 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:34.090789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:34.090996 (+ 207us) server_negotiation.cc:407] Connection header received
0929 11:26:34.091452 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:34.091457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:34.091536 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:34.091625 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:34.092034 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.092780 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.095223 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.095924 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.096061 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:34.096652 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:34.096775 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:34.096851 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:34.097069 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 11:26:34.097147 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 11:26:34.097645 25833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:34.090608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:37411 (local address 127.0.0.1:58470)
0929 11:26:34.090868 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:34.090889 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:34.091116 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:34.091681 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:34.091686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:34.091701 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:26:34.091902 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:34.091914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.093010 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.093018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:34.094967 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:34.094995 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.096113 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.096120 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:34.096339 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:34.096433 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:26:34.096881 (+ 448us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:26:34.096922 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 11:26:34.097016 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 11:26:34.097203 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 11:26:34.102222 25448 raft_consensus.cc:1261] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f [term 1 FOLLOWER]: Refusing update from remote peer 59496649db32403cb1065957b5768435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:34.102257 25587 raft_consensus.cc:1261] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf [term 1 FOLLOWER]: Refusing update from remote peer 59496649db32403cb1065957b5768435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:34.102949 25838 consensus_queue.cc:1035] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:34.103452 25813 consensus_queue.cc:1035] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:34.108371 25802 tablet_replica.cc:439] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f: TxnStatusTablet state changed. Reason: New leader 59496649db32403cb1065957b5768435. Latest consensus state: current_term: 1 leader_uuid: "59496649db32403cb1065957b5768435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } }
I20260929 11:26:34.108675 25802 tablet_replica.cc:442] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:26:34.112607 25838 tablet_replica.cc:439] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59496649db32403cb1065957b5768435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } }
I20260929 11:26:34.112941 25838 tablet_replica.cc:442] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:26:34.116484 25803 tablet_replica.cc:439] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf: TxnStatusTablet state changed. Reason: New leader 59496649db32403cb1065957b5768435. Latest consensus state: current_term: 1 leader_uuid: "59496649db32403cb1065957b5768435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } }
I20260929 11:26:34.116832 25803 tablet_replica.cc:442] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:26:34.118046 25813 tablet_replica.cc:439] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "59496649db32403cb1065957b5768435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } }
I20260929 11:26:34.118356 25813 tablet_replica.cc:442] T d50056b2f1bb4058aee120243cb5538d P 59496649db32403cb1065957b5768435: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:26:34.119212 25802 tablet_replica.cc:439] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59496649db32403cb1065957b5768435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } }
I20260929 11:26:34.119460 25802 tablet_replica.cc:442] T d50056b2f1bb4058aee120243cb5538d P fe6ba3ca201a4969b10f93862746aa7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:26:34.122109 25844 mvcc.cc:205] Tried to move back new op lower bound from 7334630171036192768 to 7334630170808758272. Current Snapshot: MvccSnapshot[applied={T|T < 7334630171036192768}]
I20260929 11:26:34.131597 25803 tablet_replica.cc:439] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "59496649db32403cb1065957b5768435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 } } peers { permanent_uuid: "b22b2717539348d984d5014d1909cecf" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39003 } } peers { permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 } } }
I20260929 11:26:34.135205 25803 tablet_replica.cc:442] T d50056b2f1bb4058aee120243cb5538d P b22b2717539348d984d5014d1909cecf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:26:34.144228 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:34.121489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.23.153.2:39003)
0929 11:26:34.135847 (+ 14358us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:34.135854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:34.138479 (+ 2625us) server_negotiation.cc:407] Connection header received
0929 11:26:34.138692 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:34.138697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:34.138779 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:34.138942 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:34.139524 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.140323 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.141784 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.142242 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.142359 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:34.142841 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:34.143509 (+ 668us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:34.143728 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:34.143789 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 11:26:34.143879 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14218,"thread_start_us":16231,"threads_started":1}
I20260929 11:26:34.162189 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:34.155827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:37411 (local address 127.23.153.2:59215)
0929 11:26:34.156515 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:34.156531 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:34.156694 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:34.157021 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:34.157025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:34.157037 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:26:34.157367 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:34.157376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.158416 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.158421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:34.159759 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:34.159778 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.160988 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.160992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:34.161134 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:34.161203 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:26:34.161666 (+ 463us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:26:34.161675 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 11:26:34.161745 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 11:26:34.161880 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":373,"threads_started":1}
I20260929 11:26:34.162207 25794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:34.155828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:59215 (local address 127.23.153.3:37411)
0929 11:26:34.156029 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:34.156035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:34.156627 (+ 592us) server_negotiation.cc:407] Connection header received
0929 11:26:34.156847 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:34.156851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:34.156919 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:34.156995 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:34.157517 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.158276 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.159948 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.160768 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.160936 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:34.161391 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:34.161566 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:34.161674 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:34.161769 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 11:26:34.161858 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:26:34.166692 25803 consensus_queue.cc:1035] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe6ba3ca201a4969b10f93862746aa7f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:34.174700 25727 raft_consensus.cc:1261] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435 [term 1 FOLLOWER]: Refusing update from remote peer b22b2717539348d984d5014d1909cecf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:34.176000 25803 consensus_queue.cc:1035] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [LEADER]: Connected to new peer: Peer: permanent_uuid: "59496649db32403cb1065957b5768435" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 37411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:34.224938 25862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:34.217928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:39003 (local address 127.23.153.3:33811)
0929 11:26:34.218817 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:34.218836 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:34.219004 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:34.219462 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:34.219467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:34.219480 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:26:34.219926 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:34.219937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.221180 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.221184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:34.222448 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:34.222468 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.223527 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.223531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:34.223703 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:34.223796 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:26:34.224209 (+ 413us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:26:34.224222 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 11:26:34.224342 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 11:26:34.224502 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":760,"threads_started":1}
I20260929 11:26:34.225029 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:34.218631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:33811 (local address 127.23.153.2:39003)
0929 11:26:34.218775 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:34.218781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:34.218923 (+ 142us) server_negotiation.cc:407] Connection header received
0929 11:26:34.219239 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:34.219245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:34.219342 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:34.219437 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:34.220096 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.221032 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.222651 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.223206 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.223305 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:34.223970 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:34.224103 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:34.224193 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:34.224634 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 11:26:34.224720 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 11:26:34.268016 25290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:26:34.268338 25290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:26:34.272246 25290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b0ee9ea79cc4f788b552e6cecfb7124: Sending DeleteTablet for 3 replicas of tablet 124551c5dfaa4ae58a1367da9461b714
I20260929 11:26:34.273360 25427 tablet_service.cc:1514] Processing DeleteTablet for tablet 124551c5dfaa4ae58a1367da9461b714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:26:34 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:43584
I20260929 11:26:34.273629 25567 tablet_service.cc:1514] Processing DeleteTablet for tablet 124551c5dfaa4ae58a1367da9461b714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:26:34 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:41382
I20260929 11:26:34.274369 25706 tablet_service.cc:1514] Processing DeleteTablet for tablet 124551c5dfaa4ae58a1367da9461b714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:26:34 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:58458
I20260929 11:26:34.288784 25866 tablet_replica.cc:330] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf: stopping tablet replica
I20260929 11:26:34.289069 25866 raft_consensus.cc:2229] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:26:34.289733 25866 raft_consensus.cc:2258] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:26:34.291142 25866 ts_tablet_manager.cc:1905] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:26:34.296727 25867 tablet_replica.cc:330] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435: stopping tablet replica
I20260929 11:26:34.297194 25867 raft_consensus.cc:2229] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:26:34.297593 25867 raft_consensus.cc:2258] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:26:34.298255 25865 tablet_replica.cc:330] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f: stopping tablet replica
I20260929 11:26:34.299233 25866 ts_tablet_manager.cc:1918] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:26:34.299388 25866 log.cc:1196] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/124551c5dfaa4ae58a1367da9461b714
I20260929 11:26:34.299764 25865 raft_consensus.cc:2229] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:26:34.299885 25866 ts_tablet_manager.cc:1939] T 124551c5dfaa4ae58a1367da9461b714 P b22b2717539348d984d5014d1909cecf: Deleting consensus metadata
I20260929 11:26:34.300161 25865 raft_consensus.cc:2258] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:26:34.299265 25867 ts_tablet_manager.cc:1905] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:26:34.301111 25276 catalog_manager.cc:4841] TS b22b2717539348d984d5014d1909cecf (127.23.153.2:39003): tablet 124551c5dfaa4ae58a1367da9461b714 (table test-table [id=aeadf017d1544b118c9939c4c06bfc27]) successfully deleted
I20260929 11:26:34.306047 25865 ts_tablet_manager.cc:1905] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:26:34.315142 25865 ts_tablet_manager.cc:1918] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
Sep 29 11:26:34 dist-test-slave-qk05 krb5kdc[25246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681192, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:26:34.319553 25867 ts_tablet_manager.cc:1918] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:26:34.319862 25867 log.cc:1196] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/124551c5dfaa4ae58a1367da9461b714
I20260929 11:26:34.320457 25867 ts_tablet_manager.cc:1939] T 124551c5dfaa4ae58a1367da9461b714 P 59496649db32403cb1065957b5768435: Deleting consensus metadata
I20260929 11:26:34.321853 25278 catalog_manager.cc:4841] TS 59496649db32403cb1065957b5768435 (127.23.153.3:37411): tablet 124551c5dfaa4ae58a1367da9461b714 (table test-table [id=aeadf017d1544b118c9939c4c06bfc27]) successfully deleted
I20260929 11:26:34.322245 25865 log.cc:1196] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/124551c5dfaa4ae58a1367da9461b714
I20260929 11:26:34.322906 25865 ts_tablet_manager.cc:1939] T 124551c5dfaa4ae58a1367da9461b714 P fe6ba3ca201a4969b10f93862746aa7f: Deleting consensus metadata
I20260929 11:26:34.324452 25278 catalog_manager.cc:4841] TS fe6ba3ca201a4969b10f93862746aa7f (127.23.153.1:45357): tablet 124551c5dfaa4ae58a1367da9461b714 (table test-table [id=aeadf017d1544b118c9939c4c06bfc27]) successfully deleted
I20260929 11:26:34.326323 25869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:34.281694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.23.153.1:45357)
0929 11:26:34.301167 (+ 19473us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:34.301173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:34.301203 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:26:34.301272 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:34.301277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:34.301388 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:34.301878 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:34.303944 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.305200 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.307321 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.308034 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.308169 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:34.317351 (+ 9182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:34.317374 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:34.317400 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:34.317455 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:34.321572 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:34.322688 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:34.322700 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:34.322704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:34.322957 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:34.323474 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:34.323480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:34.323484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:34.324062 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:34.324367 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:34.325658 (+ 1291us) server_negotiation.cc:299] Negotiation successful
0929 11:26:34.325924 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19321,"thread_start_us":20001,"threads_started":1}
W20260929 11:26:34.327212 25493 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43594
I20260929 11:26:34.349972 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:34.332550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36728 (local address 127.23.153.62:45917)
0929 11:26:34.332824 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:34.332832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:34.333189 (+ 357us) server_negotiation.cc:407] Connection header received
0929 11:26:34.333375 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:34.333381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:34.333465 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:34.333570 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:34.335426 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.336629 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.338520 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.339257 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.339404 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:34.342766 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:34.342773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:34.342779 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:34.342864 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:34.346614 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:34.347714 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:34.347720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:34.347724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:34.347901 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:34.348334 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:34.348340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:34.348343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:34.348750 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:34.349037 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:34.349355 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 11:26:34.349608 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260929 11:26:34.350616 25290 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36728
I20260929 11:26:34.353209 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 25362
I20260929 11:26:34.373227 25869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:34.354607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43602 (local address 127.23.153.1:45357)
0929 11:26:34.355534 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:34.355540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:34.355633 (+ 93us) server_negotiation.cc:407] Connection header received
0929 11:26:34.355897 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:34.355903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:34.355986 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:34.356090 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:34.357631 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.358713 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.360683 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.361411 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.361571 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:34.364655 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:34.364662 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:34.364667 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:34.364718 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:34.368930 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:34.369948 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:34.369955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:34.369959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:34.370160 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:34.371128 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:34.371135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:34.371138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:34.371523 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:34.371807 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:34.372572 (+ 765us) server_negotiation.cc:299] Negotiation successful
0929 11:26:34.372882 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780}
W20260929 11:26:34.375046 25493 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43602
W20260929 11:26:34.379715 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 25362: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:26:34.379989 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 25362
I20260929 11:26:34.403129 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 25502
Sep 29 11:26:34 dist-test-slave-qk05 krb5kdc[25246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681192, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.153.2@KRBTEST.COM
I20260929 11:26:34.427048 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:34.405597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41406 (local address 127.23.153.2:39003)
0929 11:26:34.405840 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:34.405844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:34.405862 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:26:34.405916 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:34.405920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:34.405986 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:34.406109 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:34.408011 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.409330 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.411501 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.412223 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.412370 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:34.418406 (+ 6036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:34.418434 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:34.418461 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:34.418521 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:34.423107 (+ 4586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:34.424193 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:34.424205 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:34.424209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:34.424443 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:34.424935 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:34.424940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:34.424948 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:34.425457 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:34.425874 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:34.426141 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 11:26:34.426411 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 11:26:34.428158 25632 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41406
W20260929 11:26:34.429013 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 25502: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:26:34.429219 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 25502
I20260929 11:26:34.445230 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 25641
Sep 29 11:26:34 dist-test-slave-qk05 krb5kdc[25246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681192, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.153.3@KRBTEST.COM
I20260929 11:26:34.464490 25794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:34.446048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58484 (local address 127.23.153.3:37411)
0929 11:26:34.446240 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:34.446246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:34.446269 (+ 23us) server_negotiation.cc:407] Connection header received
0929 11:26:34.446399 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:34.446405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:34.446492 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:34.446625 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:34.448552 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.449886 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.451701 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:34.452512 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:34.452675 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:34.457281 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:34.457301 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:34.457320 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:34.457456 (+ 136us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:34.460641 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:34.461692 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:34.461700 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:34.461702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:34.461846 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:34.462320 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:34.462324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:34.462326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:34.462735 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:34.463005 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:34.463585 (+ 580us) server_negotiation.cc:299] Negotiation successful
0929 11:26:34.463916 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 11:26:34.465306 25771 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58484
W20260929 11:26:34.466069 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 25641: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:26:34.466254 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 25641
I20260929 11:26:34.486256 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 25259
I20260929 11:26:34.487571 25331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:34.603499 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 25259
2026-09-29T11:26:34Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5135 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:26:34 dist-test-slave-qk05 krb5kdc[25877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:26:34 dist-test-slave-qk05 krb5kdc[25877](info): set up 2 sockets
Sep 29 11:26:34 dist-test-slave-qk05 krb5kdc[25877](info): commencing operation
krb5kdc: starting...
W20260929 11:26:35.627424 24164 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:26:35 dist-test-slave-qk05 krb5kdc[25877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:26:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:26:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:26:35.814610 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:38829
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:38829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:36.158895 25890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:36.159299 25890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:36.159441 25890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:36.166880 25890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:36.167050 25890 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 11:26:36.167114 25890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:36.167201 25890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:36.167270 25890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:36.167354 25890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:36.177436 25890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:38829
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:38829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:36.178755 25890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:36.180171 25890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:36.192299 25895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:36.192278 25898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:36.192252 25896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:36.193656 25890 server_base.cc:1034] running on GCE node
I20260929 11:26:36.194491 25890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:36.196233 25890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:36.197492 25890 hybrid_clock.cc:648] HybridClock initialized: now 1790681196197409 us; error 57 us; skew 500 ppm
I20260929 11:26:36.197847 25890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:36 dist-test-slave-qk05 krb5kdc[25877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681196, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:36.201151 25890 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:26:36.202950 25890 webserver.cc:460] Webserver started at http://127.23.153.62:38697/ using document root <none> and password file <none>
I20260929 11:26:36.203764 25890 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:36.203874 25890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:36.204200 25890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:36.207495 25890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "05b1fa39fce34f758dd95f2337655edb"
format_stamp: "Formatted at 2026-09-29 11:26:36 on dist-test-slave-qk05"
server_key: "d78a46ec37facb476099b1cef44321b6"
server_key_iv: "92eafc0e0020bd2cc281f2144d9d227d"
server_key_version: "encryptionkey@0"
I20260929 11:26:36.208401 25890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "05b1fa39fce34f758dd95f2337655edb"
format_stamp: "Formatted at 2026-09-29 11:26:36 on dist-test-slave-qk05"
server_key: "d78a46ec37facb476099b1cef44321b6"
server_key_iv: "92eafc0e0020bd2cc281f2144d9d227d"
server_key_version: "encryptionkey@0"
I20260929 11:26:36.214370 25890 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260929 11:26:36.218567 25905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:36.220116 25890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 11:26:36.220335 25890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "05b1fa39fce34f758dd95f2337655edb"
format_stamp: "Formatted at 2026-09-29 11:26:36 on dist-test-slave-qk05"
server_key: "d78a46ec37facb476099b1cef44321b6"
server_key_iv: "92eafc0e0020bd2cc281f2144d9d227d"
server_key_version: "encryptionkey@0"
I20260929 11:26:36.220582 25890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:36.243868 25890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:36.248494 25890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:36.248777 25890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:36.270664 25890 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:38829
I20260929 11:26:36.270684 25966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:38829 every 8 connection(s)
I20260929 11:26:36.272187 25890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:26:36.273887 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 25890
I20260929 11:26:36.274073 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:26:36.274716 24164 external_mini_cluster.cc:1428] Setting key fda06cc61dd0e16d4ab39be4de690b9c
I20260929 11:26:36.276966 25967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 11:26:36 dist-test-slave-qk05 krb5kdc[25877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:36.291937 25967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb: Bootstrap starting.
I20260929 11:26:36.295619 25967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:36.296911 25967 log.cc:824] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:36.300139 25967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb: No bootstrap required, opened a new log
I20260929 11:26:36.300904 25970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:36.276936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.23.153.62:38829)
0929 11:26:36.278109 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:36.278123 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:36.278170 (+ 47us) server_negotiation.cc:407] Connection header received
0929 11:26:36.279957 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:36.279987 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:36.280495 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:36.280921 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:36.282823 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:36.284515 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:36.286561 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:36.287423 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:36.287585 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:36.292714 (+ 5129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:36.292724 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:36.292746 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:36.292799 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:36.296269 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:36.297260 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:36.297268 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:36.297275 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:36.297466 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:36.297882 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:36.297886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:36.297889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:36.298489 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:36.298783 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:36.299292 (+ 509us) server_negotiation.cc:299] Negotiation successful
0929 11:26:36.299752 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":926,"thread_start_us":685,"threads_started":1}
I20260929 11:26:36.309026 25967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b1fa39fce34f758dd95f2337655edb" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38829 } }
I20260929 11:26:36.309571 25967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:36.309689 25967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b1fa39fce34f758dd95f2337655edb, State: Initialized, Role: FOLLOWER
I20260929 11:26:36.310443 25967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b1fa39fce34f758dd95f2337655edb" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38829 } }
I20260929 11:26:36.310698 25967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:26:36.310830 25967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:26:36.310989 25967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:36.313382 25967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b1fa39fce34f758dd95f2337655edb" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38829 } }
I20260929 11:26:36.314036 25967 leader_election.cc:302] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05b1fa39fce34f758dd95f2337655edb; no voters:
I20260929 11:26:36.314697 25967 leader_election.cc:288] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:26:36.314805 25972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:36.315982 25972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb [term 1 LEADER]: Becoming Leader. State: Replica: 05b1fa39fce34f758dd95f2337655edb, State: Running, Role: LEADER
I20260929 11:26:36.316584 25972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b1fa39fce34f758dd95f2337655edb" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38829 } }
I20260929 11:26:36.317003 25967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:26:36.319762 25974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05b1fa39fce34f758dd95f2337655edb. Latest consensus state: current_term: 1 leader_uuid: "05b1fa39fce34f758dd95f2337655edb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b1fa39fce34f758dd95f2337655edb" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38829 } } }
I20260929 11:26:36.320200 25974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:36.320693 25973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05b1fa39fce34f758dd95f2337655edb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05b1fa39fce34f758dd95f2337655edb" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38829 } } }
I20260929 11:26:36.320973 25973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:36.322897 25979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:26:36.328893 25979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:26:36.339962 25979 catalog_manager.cc:1347] Generated new cluster ID: d0869fb6fae1493a91be73c55b35d0b8
I20260929 11:26:36.340113 25979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:26:36.366665 25979 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:26:36.368757 25979 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:26:36.375990 25979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb: Generated new TSK 0
I20260929 11:26:36.376959 25979 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
I20260929 11:26:36.439270 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38829
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:36.832152 25993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:36.832471 25993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:36.832574 25993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:36.842089 25993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:36.842432 25993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:36.842602 25993 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 11:26:36.842661 25993 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:26:36.842747 25993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:26:36.853781 25993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.153.62:38829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:36.855413 25993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:36.856830 25993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:36.870615 25998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:36.870661 25999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:36.870602 26001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:36.871964 25993 server_base.cc:1034] running on GCE node
I20260929 11:26:36.872673 25993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:36.873687 25993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:36.874957 25993 hybrid_clock.cc:648] HybridClock initialized: now 1790681196874898 us; error 43 us; skew 500 ppm
I20260929 11:26:36.875365 25993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:36 dist-test-slave-qk05 krb5kdc[25877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681196, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:36.878819 25993 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:26:36.880960 25993 webserver.cc:460] Webserver started at http://127.23.153.1:42239/ using document root <none> and password file <none>
I20260929 11:26:36.881716 25993 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:36.881845 25993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:36.882162 25993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:36.885460 25993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "77cf22706b4f487c9d66a93f9bc02804"
format_stamp: "Formatted at 2026-09-29 11:26:36 on dist-test-slave-qk05"
server_key: "4680233577e360ff98d8d376bd6c8410"
server_key_iv: "00b2eaaf0614a71385fd065a3a3669f4"
server_key_version: "encryptionkey@0"
I20260929 11:26:36.886341 25993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "77cf22706b4f487c9d66a93f9bc02804"
format_stamp: "Formatted at 2026-09-29 11:26:36 on dist-test-slave-qk05"
server_key: "4680233577e360ff98d8d376bd6c8410"
server_key_iv: "00b2eaaf0614a71385fd065a3a3669f4"
server_key_version: "encryptionkey@0"
I20260929 11:26:36.892339 25993 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 11:26:36.896409 26008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:36.898108 25993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 11:26:36.898307 25993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "77cf22706b4f487c9d66a93f9bc02804"
format_stamp: "Formatted at 2026-09-29 11:26:36 on dist-test-slave-qk05"
server_key: "4680233577e360ff98d8d376bd6c8410"
server_key_iv: "00b2eaaf0614a71385fd065a3a3669f4"
server_key_version: "encryptionkey@0"
I20260929 11:26:36.898530 25993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:36.914264 25993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:36.918833 25993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:36.919165 25993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:36.922096 25993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:36.922240 25993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:36.922369 25993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:36.922465 25993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:26:36 dist-test-slave-qk05 krb5kdc[25877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681196, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:36.947317 26017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:36.922647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:53383 (local address 127.23.153.62:38829)
0929 11:26:36.925590 (+ 2943us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:36.925597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:36.925621 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:26:36.927557 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:36.927566 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:36.927691 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:36.927836 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:36.930350 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:36.931625 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:36.934451 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:36.935177 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:36.935364 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:36.940322 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:36.940331 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:36.940336 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:36.940386 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:36.943151 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:36.944620 (+ 1469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:36.944627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:36.944630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:36.944796 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:36.945318 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:36.945324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:36.945328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:36.945622 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:36.945767 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:36.946608 (+ 841us) server_negotiation.cc:299] Negotiation successful
0929 11:26:36.946896 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2810,"thread_start_us":2690,"threads_started":1}
I20260929 11:26:36.948627 26018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:36.923496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:38829 (local address 127.23.153.1:53383)
0929 11:26:36.924777 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:36.924847 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:36.927115 (+ 2268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:36.928147 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:36.928157 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:36.929092 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:36.930040 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:36.930070 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:36.931891 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:36.931897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:36.934058 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:36.934107 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:36.935485 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:36.935501 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:36.935755 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:36.936613 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:36.936650 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:36.940011 (+ 3361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:36.943490 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:36.943502 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:36.943530 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:36.944407 (+ 877us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:36.944933 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:36.944937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:36.944940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:36.945190 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:36.945787 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:36.945797 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:36.946197 (+ 400us) client_negotiation.cc:769] Sending connection context
0929 11:26:36.946474 (+ 277us) client_negotiation.cc:240] Negotiation successful
0929 11:26:36.947109 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":417,"threads_started":1}
I20260929 11:26:36.974020 25993 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:45795
I20260929 11:26:36.974033 26128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:45795 every 8 connection(s)
I20260929 11:26:36.975886 25993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:26:36.978094 26129 heartbeater.cc:347] Connected to a master server at 127.23.153.62:38829
I20260929 11:26:36.978346 26129 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:36.979094 26129 heartbeater.cc:511] Master 127.23.153.62:38829 requested a full tablet report, sending...
I20260929 11:26:36.981436 25922 ts_manager.cc:194] Registered new tserver with Master: 77cf22706b4f487c9d66a93f9bc02804 (127.23.153.1:45795)
I20260929 11:26:36.981985 26017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:36.966322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:37005 (local address 127.23.153.62:38829)
0929 11:26:36.966538 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:36.966546 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:36.967166 (+ 620us) server_negotiation.cc:407] Connection header received
0929 11:26:36.967379 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:36.967383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:36.967452 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:36.967541 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:36.968940 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:36.969660 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:36.971725 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:36.972261 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:36.972646 (+ 385us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:36.975619 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:36.975693 (+ 74us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:36.975698 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:36.975743 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:36.978947 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:36.979873 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:36.979878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:36.979881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:36.980040 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:36.980449 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:36.980455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:36.980462 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:36.980781 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:36.980923 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:36.981371 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 11:26:36.981606 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 11:26:36.982018 26124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:36.966289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:38829 (local address 127.23.153.1:37005)
0929 11:26:36.967001 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:36.967057 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:36.967260 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:36.967600 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:36.967605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:36.968302 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:36.968713 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:36.968729 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:36.969810 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:36.969815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:36.971520 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:36.971544 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:36.972694 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:36.972706 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:36.972947 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:36.973949 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:36.973973 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:36.975368 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:36.979141 (+ 3773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:36.979147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:36.979151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:36.979704 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:36.980183 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:36.980187 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:36.980190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:36.980328 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:36.980909 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:36.980913 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:36.981093 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 11:26:36.981211 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 11:26:36.981573 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":343,"threads_started":1}
I20260929 11:26:36.982602 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 25993
I20260929 11:26:36.982775 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:26:36.983299 25922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:53383
I20260929 11:26:36.983489 24164 external_mini_cluster.cc:1428] Setting key 6caa091f5dc94ad5b2f2f95c9746ae3a
WARNING: no policy specified for kudu/127.23.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
I20260929 11:26:37.022262 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38829
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:37.389150 26133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:37.389456 26133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:37.389588 26133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:37.397320 26133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:37.397459 26133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:37.397543 26133 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 11:26:37.397601 26133 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:26:37.397671 26133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:26:37.408828 26133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.153.62:38829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:37.410248 26133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:37.411731 26133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:37.426190 26138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:37.426249 26141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:37.426224 26139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:37.426939 26133 server_base.cc:1034] running on GCE node
I20260929 11:26:37.427773 26133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:37.428823 26133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:37.430054 26133 hybrid_clock.cc:648] HybridClock initialized: now 1790681197429993 us; error 56 us; skew 500 ppm
I20260929 11:26:37.430424 26133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:37 dist-test-slave-qk05 krb5kdc[25877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681197, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:37.433954 26133 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:26:37.435652 26133 webserver.cc:460] Webserver started at http://127.23.153.2:33211/ using document root <none> and password file <none>
I20260929 11:26:37.436404 26133 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:37.436535 26133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:37.436870 26133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:37.440102 26133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c"
format_stamp: "Formatted at 2026-09-29 11:26:37 on dist-test-slave-qk05"
server_key: "96b50ab20625b3fd6f2d95bebf1a88f5"
server_key_iv: "0484e8859c57e5506c982888599f62e7"
server_key_version: "encryptionkey@0"
I20260929 11:26:37.440910 26133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c"
format_stamp: "Formatted at 2026-09-29 11:26:37 on dist-test-slave-qk05"
server_key: "96b50ab20625b3fd6f2d95bebf1a88f5"
server_key_iv: "0484e8859c57e5506c982888599f62e7"
server_key_version: "encryptionkey@0"
I20260929 11:26:37.446702 26133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 11:26:37.450912 26148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:37.452596 26133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:26:37.452795 26133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c"
format_stamp: "Formatted at 2026-09-29 11:26:37 on dist-test-slave-qk05"
server_key: "96b50ab20625b3fd6f2d95bebf1a88f5"
server_key_iv: "0484e8859c57e5506c982888599f62e7"
server_key_version: "encryptionkey@0"
I20260929 11:26:37.453015 26133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:37.466467 26133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:37.470664 26133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:37.470961 26133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:37.473978 26133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:37.474100 26133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:37.474221 26133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:37.474294 26133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:26:37 dist-test-slave-qk05 krb5kdc[25877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681197, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:37.500525 26017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:37.474849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:45265 (local address 127.23.153.62:38829)
0929 11:26:37.475079 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:37.475086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:37.476665 (+ 1579us) server_negotiation.cc:407] Connection header received
0929 11:26:37.479826 (+ 3161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:37.479833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:37.479944 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:37.480079 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:37.482725 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:37.483532 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:37.486718 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:37.487365 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:37.487510 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:37.493062 (+ 5552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:37.493071 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:37.493077 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:37.493131 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:37.496516 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:37.497734 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:37.497740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:37.497742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:37.497885 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:37.498484 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:37.498489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:37.498491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:37.498825 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:37.498948 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:37.500060 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0929 11:26:37.500270 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 11:26:37.502606 26157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:37.475535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:38829 (local address 127.23.153.2:45265)
0929 11:26:37.476432 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:37.476486 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:37.479367 (+ 2881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:37.480507 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:37.480527 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:37.481411 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:37.482490 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:37.482520 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:37.483740 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:37.483747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:37.486454 (+ 2707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:37.486481 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:37.487834 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:37.487842 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:37.488138 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:37.488971 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:37.489015 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:37.492651 (+ 3636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:37.496750 (+ 4099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:37.496764 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:37.496789 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:37.497550 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:37.498004 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:37.498011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:37.498015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:37.498343 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:37.498967 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:37.498977 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:37.499573 (+ 596us) client_negotiation.cc:769] Sending connection context
0929 11:26:37.499941 (+ 368us) client_negotiation.cc:240] Negotiation successful
0929 11:26:37.500614 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":346,"threads_started":1}
I20260929 11:26:37.530613 26133 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:45881
I20260929 11:26:37.530613 26267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:45881 every 8 connection(s)
I20260929 11:26:37.532251 26133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:26:37.532850 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 26133
I20260929 11:26:37.533032 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:26:37.533607 24164 external_mini_cluster.cc:1428] Setting key bc9f20982c0f99d74507bf949530a2df
I20260929 11:26:37.534950 26268 heartbeater.cc:347] Connected to a master server at 127.23.153.62:38829
I20260929 11:26:37.535722 26268 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:37.536586 26268 heartbeater.cc:511] Master 127.23.153.62:38829 requested a full tablet report, sending...
I20260929 11:26:37.538167 25922 ts_manager.cc:194] Registered new tserver with Master: 2e46c2a7ac0341a4acd1de07bb4cbf4c (127.23.153.2:45881)
I20260929 11:26:37.539247 25922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:45265
I20260929 11:26:37.541356 26017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:37.519729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:58073 (local address 127.23.153.62:38829)
0929 11:26:37.519926 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:37.519932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:37.521424 (+ 1492us) server_negotiation.cc:407] Connection header received
0929 11:26:37.521750 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:37.521758 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:37.521845 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:37.521924 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:37.524460 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:37.525627 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:37.528450 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:37.529041 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:37.529201 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:37.531895 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:37.531903 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:37.531907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:37.531949 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:37.536236 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:37.537168 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:37.537174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:37.537178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:37.537367 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:37.539684 (+ 2317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:37.539690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:37.539694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:37.540137 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:37.540307 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:37.540811 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 11:26:37.541053 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:26:37.541498 26248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:37.520085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:38829 (local address 127.23.153.2:58073)
0929 11:26:37.521268 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:37.521299 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:37.521567 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:37.522181 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:37.522185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:37.522901 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:37.524086 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:37.524109 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:37.525939 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:37.525945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:37.528100 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:37.528129 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:37.529497 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:37.529505 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:37.529694 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:37.530734 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:37.530765 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:37.531635 (+ 870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:37.536424 (+ 4789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:37.536430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:37.536434 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:37.536979 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:37.537490 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:37.537496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:37.537500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:37.537680 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:37.540280 (+ 2600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:37.540286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:37.540511 (+ 225us) client_negotiation.cc:769] Sending connection context
0929 11:26:37.540692 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 11:26:37.541059 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"thread_start_us":859,"threads_started":1}
WARNING: no policy specified for kudu/127.23.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
I20260929 11:26:37.576020 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38829
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:37.925180 26272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:37.925475 26272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:37.925613 26272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:37.933535 26272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:37.933682 26272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:37.933811 26272 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 11:26:37.933876 26272 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:26:37.933976 26272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:26:37.944958 26272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.153.62:38829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:37.946427 26272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:37.947887 26272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:37.961759 26280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:37.961711 26277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:37.961781 26278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:37.962584 26272 server_base.cc:1034] running on GCE node
I20260929 11:26:37.963624 26272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:37.964922 26272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:37.966163 26272 hybrid_clock.cc:648] HybridClock initialized: now 1790681197966104 us; error 47 us; skew 500 ppm
I20260929 11:26:37.966507 26272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:37 dist-test-slave-qk05 krb5kdc[25877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681197, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:37.969892 26272 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:26:37.971771 26272 webserver.cc:460] Webserver started at http://127.23.153.3:46349/ using document root <none> and password file <none>
I20260929 11:26:37.972527 26272 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:37.972646 26272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:37.972949 26272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:37.976184 26272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "390b6ee45e10487992567c61f5971511"
format_stamp: "Formatted at 2026-09-29 11:26:37 on dist-test-slave-qk05"
server_key: "603762c6f6e9afbe6c340b18c1d472b4"
server_key_iv: "cc5fb367f950808424b36321729ca626"
server_key_version: "encryptionkey@0"
I20260929 11:26:37.977010 26272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "390b6ee45e10487992567c61f5971511"
format_stamp: "Formatted at 2026-09-29 11:26:37 on dist-test-slave-qk05"
server_key: "603762c6f6e9afbe6c340b18c1d472b4"
server_key_iv: "cc5fb367f950808424b36321729ca626"
server_key_version: "encryptionkey@0"
I20260929 11:26:37.983253 26272 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 11:26:37.986960 26129 heartbeater.cc:503] Master 127.23.153.62:38829 was elected leader, sending a full tablet report...
I20260929 11:26:37.987478 26287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:37.989789 26272 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 11:26:37.990025 26272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "390b6ee45e10487992567c61f5971511"
format_stamp: "Formatted at 2026-09-29 11:26:37 on dist-test-slave-qk05"
server_key: "603762c6f6e9afbe6c340b18c1d472b4"
server_key_iv: "cc5fb367f950808424b36321729ca626"
server_key_version: "encryptionkey@0"
I20260929 11:26:37.990327 26272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:38.013522 26272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:38.018963 26272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:38.019323 26272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:38.022414 26272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:38.022534 26272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:38.022658 26272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:38.022800 26272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 29 11:26:38 dist-test-slave-qk05 krb5kdc[25877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681197, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:38.048352 26017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.023083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:58595 (local address 127.23.153.62:38829)
0929 11:26:38.023260 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.023264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.025016 (+ 1752us) server_negotiation.cc:407] Connection header received
0929 11:26:38.028250 (+ 3234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.028256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.028355 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.028463 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:38.031795 (+ 3332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.033192 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.035146 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.035792 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.035948 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.041475 (+ 5527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:38.041483 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:38.041488 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:38.041542 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:38.044794 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:38.045668 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:38.045674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:38.045678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:38.045823 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:38.046317 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:38.046323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:38.046326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:38.046642 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:38.046789 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.047713 (+ 924us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.047979 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 11:26:38.049938 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.023754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:38829 (local address 127.23.153.3:58595)
0929 11:26:38.024815 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:38.024868 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:38.027709 (+ 2841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:38.028844 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:38.028857 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:38.029911 (+ 1054us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:38.031397 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.031438 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.033422 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.033428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.034868 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.034915 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.036122 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.036131 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.036495 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.037310 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:38.037344 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:38.041061 (+ 3717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:38.044985 (+ 3924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:38.044997 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:38.045017 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:38.045509 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:38.045945 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:38.045950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:38.045954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:38.046184 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:38.046780 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:38.046789 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:38.047273 (+ 484us) client_negotiation.cc:769] Sending connection context
0929 11:26:38.047564 (+ 291us) client_negotiation.cc:240] Negotiation successful
0929 11:26:38.048194 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":381,"threads_started":1}
I20260929 11:26:38.076007 26272 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:33063
I20260929 11:26:38.076133 26017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.060410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:52743 (local address 127.23.153.62:38829)
0929 11:26:38.060573 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.060578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.061197 (+ 619us) server_negotiation.cc:407] Connection header received
0929 11:26:38.061405 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.061411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.061504 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.061624 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:38.062647 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.063663 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.066038 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.066512 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.066658 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.068810 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:38.068819 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:38.068825 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:38.068876 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:38.072585 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:38.073610 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:38.073616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:38.073618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:38.073789 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:38.074271 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:38.074275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:38.074277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:38.074595 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:38.074771 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.075473 (+ 702us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.075766 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 11:26:38.076376 26378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.060396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:38829 (local address 127.23.153.3:52743)
0929 11:26:38.061041 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:38.061065 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:38.061254 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:38.061683 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:38.061688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:38.062182 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:38.062476 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.062487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.063974 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.063980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.065826 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.065846 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.066751 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.066758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.066920 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.067729 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:38.067751 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:38.068524 (+ 773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:38.072798 (+ 4274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:38.072805 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:38.072809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:38.073441 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:38.073944 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:38.073949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:38.073956 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:38.074139 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:38.074756 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:38.074762 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:38.075166 (+ 404us) client_negotiation.cc:769] Sending connection context
0929 11:26:38.075353 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 11:26:38.075809 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":307,"threads_started":1}
I20260929 11:26:38.076066 26406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:33063 every 8 connection(s)
I20260929 11:26:38.077951 26272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:26:38.080403 26407 heartbeater.cc:347] Connected to a master server at 127.23.153.62:38829
I20260929 11:26:38.080652 26407 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:38.081341 26407 heartbeater.cc:511] Master 127.23.153.62:38829 requested a full tablet report, sending...
I20260929 11:26:38.082868 25922 ts_manager.cc:194] Registered new tserver with Master: 390b6ee45e10487992567c61f5971511 (127.23.153.3:33063)
I20260929 11:26:38.083745 25922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:58595
I20260929 11:26:38.086627 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 26272
I20260929 11:26:38.086772 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:26:38.087220 24164 external_mini_cluster.cc:1428] Setting key 4a1d48ecdcc38594461e2132ebfe589e
I20260929 11:26:38.090754 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 11:26:38 dist-test-slave-qk05 krb5kdc[25877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 11:26:38 dist-test-slave-qk05 krb5kdc[25877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:38.141507 26017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.124309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.23.153.62:38829)
0929 11:26:38.124575 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.124582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.124858 (+ 276us) server_negotiation.cc:407] Connection header received
0929 11:26:38.125073 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.125078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.125347 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.125524 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:38.128143 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.129169 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.130634 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.131314 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.131548 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.135917 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:38.135923 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:38.135926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:38.135976 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:38.138577 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:38.139390 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:38.139395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:38.139397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:38.139532 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:38.139895 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:38.139899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:38.139901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:38.140230 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:38.140475 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.140910 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.141176 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 11:26:38.146281 25922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:26:38.154450 25922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:26:38.183419 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.172841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.23.153.2:45881)
0929 11:26:38.174381 (+ 1540us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.174393 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.175409 (+ 1016us) server_negotiation.cc:407] Connection header received
0929 11:26:38.175943 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.175953 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.176275 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.176507 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:38.177509 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.178464 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.180527 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.182027 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.182179 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.182381 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.182801 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.182991 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1338,"thread_start_us":1157,"threads_started":1}
I20260929 11:26:38.183468 26425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.173236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:45881 (local address 127.0.0.1:49070)
0929 11:26:38.175247 (+ 2011us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:38.175271 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:38.175449 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:38.176760 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:38.176765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:38.176792 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:38.177305 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.177316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.178632 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.178636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.180265 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.180297 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.182252 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.182259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.182493 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.182566 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 11:26:38.182730 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 11:26:38.182946 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1876,"mutex_wait_us":3,"thread_start_us":1676,"threads_started":1}
I20260929 11:26:38.183458 26423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.168949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.23.153.1:45795)
0929 11:26:38.170619 (+ 1670us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.170631 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.172768 (+ 2137us) server_negotiation.cc:407] Connection header received
0929 11:26:38.172885 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.172891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.173253 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.173497 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:38.175666 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.177102 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.180761 (+ 3659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.182026 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.182132 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.182367 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.182972 (+ 605us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.183190 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1466,"thread_start_us":1198,"threads_started":1}
I20260929 11:26:38.184012 26420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.168332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:45795 (local address 127.0.0.1:56608)
0929 11:26:38.172466 (+ 4134us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:38.172494 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:38.172627 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:38.173749 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:38.173755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:38.173784 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:38.175393 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.175403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.177254 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.177263 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.180563 (+ 3300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.180576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.182593 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.182599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.182779 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.182808 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 11:26:38.183652 (+ 844us) client_negotiation.cc:240] Negotiation successful
0929 11:26:38.183767 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4014,"mutex_wait_us":587,"thread_start_us":2479,"threads_started":1}
I20260929 11:26:38.184201 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.168232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:33063 (local address 127.0.0.1:38620)
0929 11:26:38.171339 (+ 3107us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:38.172228 (+ 889us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:38.172446 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:38.173720 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:38.173729 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:38.173768 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:38.175360 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.175384 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.177444 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.177449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.180282 (+ 2833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.180298 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.182538 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.182542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.182708 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.182740 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 11:26:38.182820 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 11:26:38.182975 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2731,"mutex_wait_us":192,"thread_start_us":3938,"threads_started":1}
I20260929 11:26:38.184639 26422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.168385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38620 (local address 127.23.153.3:33063)
0929 11:26:38.172146 (+ 3761us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.172156 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.172659 (+ 503us) server_negotiation.cc:407] Connection header received
0929 11:26:38.172767 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.172775 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.173123 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.173345 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:38.175693 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.177298 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.180499 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.182029 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.182179 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.182410 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.183522 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.184184 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3588,"thread_start_us":3410,"threads_started":1}
I20260929 11:26:38.188303 26198 tablet_service.cc:1467] Processing CreateTablet for tablet 39176ed2815c46759ba81f19be330353 (DEFAULT_TABLE table=test-table [id=18c86896990b4cf7b530059e37e03715]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:38.188303 26337 tablet_service.cc:1467] Processing CreateTablet for tablet 39176ed2815c46759ba81f19be330353 (DEFAULT_TABLE table=test-table [id=18c86896990b4cf7b530059e37e03715]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:38.188630 26059 tablet_service.cc:1467] Processing CreateTablet for tablet 39176ed2815c46759ba81f19be330353 (DEFAULT_TABLE table=test-table [id=18c86896990b4cf7b530059e37e03715]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:38.189929 26059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39176ed2815c46759ba81f19be330353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:38.189953 26198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39176ed2815c46759ba81f19be330353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:38.189950 26337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39176ed2815c46759ba81f19be330353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:38.203128 26426 tablet_bootstrap.cc:492] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804: Bootstrap starting.
I20260929 11:26:38.207007 26426 tablet_bootstrap.cc:654] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:38.207819 26427 tablet_bootstrap.cc:492] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Bootstrap starting.
I20260929 11:26:38.208918 26428 tablet_bootstrap.cc:492] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511: Bootstrap starting.
I20260929 11:26:38.210027 26426 log.cc:824] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:38.212517 26427 tablet_bootstrap.cc:654] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:38.212693 26428 tablet_bootstrap.cc:654] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:38.213707 26428 log.cc:824] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:38.213796 26427 log.cc:824] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:38.213922 26426 tablet_bootstrap.cc:492] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804: No bootstrap required, opened a new log
I20260929 11:26:38.214234 26426 ts_tablet_manager.cc:1397] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 11:26:38.217087 26428 tablet_bootstrap.cc:492] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511: No bootstrap required, opened a new log
I20260929 11:26:38.217370 26427 tablet_bootstrap.cc:492] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: No bootstrap required, opened a new log
I20260929 11:26:38.217391 26428 ts_tablet_manager.cc:1397] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 11:26:38.217631 26427 ts_tablet_manager.cc:1397] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 11:26:38.225185 26426 raft_consensus.cc:348] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } }
I20260929 11:26:38.225713 26426 raft_consensus.cc:374] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:38.225814 26426 raft_consensus.cc:729] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77cf22706b4f487c9d66a93f9bc02804, State: Initialized, Role: FOLLOWER
I20260929 11:26:38.226308 26427 raft_consensus.cc:348] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } }
I20260929 11:26:38.226677 26427 raft_consensus.cc:374] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:38.226521 26426 consensus_queue.cc:260] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } }
I20260929 11:26:38.226774 26427 raft_consensus.cc:729] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e46c2a7ac0341a4acd1de07bb4cbf4c, State: Initialized, Role: FOLLOWER
I20260929 11:26:38.227381 26427 consensus_queue.cc:260] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } }
I20260929 11:26:38.228437 26268 heartbeater.cc:503] Master 127.23.153.62:38829 was elected leader, sending a full tablet report...
I20260929 11:26:38.228149 26428 raft_consensus.cc:348] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } }
I20260929 11:26:38.228627 26428 raft_consensus.cc:374] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:38.228713 26428 raft_consensus.cc:729] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 390b6ee45e10487992567c61f5971511, State: Initialized, Role: FOLLOWER
I20260929 11:26:38.229213 26427 ts_tablet_manager.cc:1428] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260929 11:26:38.229368 26428 consensus_queue.cc:260] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } }
I20260929 11:26:38.235261 26426 ts_tablet_manager.cc:1428] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804: Time spent starting tablet: real 0.021s user 0.011s sys 0.009s
I20260929 11:26:38.236505 26407 heartbeater.cc:503] Master 127.23.153.62:38829 was elected leader, sending a full tablet report...
I20260929 11:26:38.237150 26428 ts_tablet_manager.cc:1428] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
I20260929 11:26:38.247258 26434 raft_consensus.cc:482] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:26:38.247570 26434 raft_consensus.cc:504] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } }
I20260929 11:26:38.249264 26434 leader_election.cc:288] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77cf22706b4f487c9d66a93f9bc02804 (127.23.153.1:45795), 2e46c2a7ac0341a4acd1de07bb4cbf4c (127.23.153.2:45881)
I20260929 11:26:38.255810 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.249535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:45881 (local address 127.23.153.3:60731)
0929 11:26:38.249845 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:38.249865 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:38.250000 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:38.250404 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:38.250409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:38.250450 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:38.250709 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.250720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.251919 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.251922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.253639 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.253656 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.254972 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.254979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.255222 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.255249 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 11:26:38.255336 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 11:26:38.255488 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":1}
I20260929 11:26:38.255800 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.249819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:60731 (local address 127.23.153.2:45881)
0929 11:26:38.250011 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.250017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.250039 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:26:38.250110 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.250115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.250189 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.250273 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:38.250860 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.251802 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.253843 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.254804 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.254972 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.255112 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.255385 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.255523 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:26:38.256469 26423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.250117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:41085 (local address 127.23.153.1:45795)
0929 11:26:38.250306 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.250312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.250670 (+ 358us) server_negotiation.cc:407] Connection header received
0929 11:26:38.250953 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.250957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.251063 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.251144 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:38.251776 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.252510 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.254596 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.255466 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.255635 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.255709 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.256088 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.256231 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:26:38.256976 26218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39176ed2815c46759ba81f19be330353" candidate_uuid: "390b6ee45e10487992567c61f5971511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" is_pre_election: true
I20260929 11:26:38.257184 26435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.249558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:45795 (local address 127.23.153.3:41085)
0929 11:26:38.250536 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:38.250557 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:38.250769 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:38.251171 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:38.251176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:38.251201 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:38.251550 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.251562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.252654 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.252660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.254408 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.254421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.255698 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.255705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.255881 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.255915 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 11:26:38.256658 (+ 743us) client_negotiation.cc:240] Negotiation successful
0929 11:26:38.256807 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":37,"thread_start_us":812,"threads_started":1}
I20260929 11:26:38.257552 26218 raft_consensus.cc:2454] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 390b6ee45e10487992567c61f5971511 in term 0.
I20260929 11:26:38.258283 26291 leader_election.cc:302] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e46c2a7ac0341a4acd1de07bb4cbf4c, 390b6ee45e10487992567c61f5971511; no voters:
I20260929 11:26:38.258198 26079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39176ed2815c46759ba81f19be330353" candidate_uuid: "390b6ee45e10487992567c61f5971511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77cf22706b4f487c9d66a93f9bc02804" is_pre_election: true
I20260929 11:26:38.258735 26434 raft_consensus.cc:2781] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:26:38.258766 26079 raft_consensus.cc:2454] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 390b6ee45e10487992567c61f5971511 in term 0.
I20260929 11:26:38.258863 26434 raft_consensus.cc:482] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:26:38.258958 26434 raft_consensus.cc:3037] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:38.261623 26434 raft_consensus.cc:504] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } }
I20260929 11:26:38.262507 26434 leader_election.cc:288] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [CANDIDATE]: Term 1 election: Requested vote from peers 77cf22706b4f487c9d66a93f9bc02804 (127.23.153.1:45795), 2e46c2a7ac0341a4acd1de07bb4cbf4c (127.23.153.2:45881)
I20260929 11:26:38.263200 26079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39176ed2815c46759ba81f19be330353" candidate_uuid: "390b6ee45e10487992567c61f5971511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77cf22706b4f487c9d66a93f9bc02804"
I20260929 11:26:38.263200 26218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39176ed2815c46759ba81f19be330353" candidate_uuid: "390b6ee45e10487992567c61f5971511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c"
I20260929 11:26:38.263454 26079 raft_consensus.cc:3037] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:38.263454 26218 raft_consensus.cc:3037] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:38.267081 26218 raft_consensus.cc:2454] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 390b6ee45e10487992567c61f5971511 in term 1.
I20260929 11:26:38.267081 26079 raft_consensus.cc:2454] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 390b6ee45e10487992567c61f5971511 in term 1.
I20260929 11:26:38.267673 26291 leader_election.cc:302] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e46c2a7ac0341a4acd1de07bb4cbf4c, 390b6ee45e10487992567c61f5971511; no voters:
I20260929 11:26:38.268028 26434 raft_consensus.cc:2781] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:38.268524 26434 raft_consensus.cc:686] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [term 1 LEADER]: Becoming Leader. State: Replica: 390b6ee45e10487992567c61f5971511, State: Running, Role: LEADER
I20260929 11:26:38.269062 26434 consensus_queue.cc:237] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } }
I20260929 11:26:38.274384 25922 catalog_manager.cc:5495] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 reported cstate change: term changed from 0 to 1, leader changed from <none> to 390b6ee45e10487992567c61f5971511 (127.23.153.3). New cstate: current_term: 1 leader_uuid: "390b6ee45e10487992567c61f5971511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } health_report { overall_health: UNKNOWN } } }
W20260929 11:26:38.285405 26269 tablet.cc:2369] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 11:26:38 dist-test-slave-qk05 krb5kdc[25877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681196, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
W20260929 11:26:38.329826 26408 tablet.cc:2369] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:26:38.336468 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.321763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:38829 (local address 127.0.0.1:54070)
0929 11:26:38.322363 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:38.322382 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:38.322544 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:38.322853 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:38.322858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:38.323625 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:38.323888 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.323899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.324995 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.325001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.325999 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.326012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.326759 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.326769 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.326924 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.327644 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:38.327663 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:38.330261 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:38.333291 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:38.333307 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:38.333314 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:38.333802 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:38.334384 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:38.334390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:38.334393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:38.334653 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:38.335466 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:38.335476 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:38.335664 (+ 188us) client_negotiation.cc:769] Sending connection context
0929 11:26:38.335829 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 11:26:38.336108 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":333,"threads_started":1}
I20260929 11:26:38.336894 26017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.321735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.23.153.62:38829)
0929 11:26:38.321981 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.321986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.322488 (+ 502us) server_negotiation.cc:407] Connection header received
0929 11:26:38.322655 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.322660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.322730 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.322810 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:38.324049 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.324868 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.326142 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.326566 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.326689 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.330505 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:38.330511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:38.330514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:38.330549 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:38.333082 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:38.333948 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:38.333954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:38.333956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:38.334118 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:38.334840 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:38.334845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:38.334847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:38.335300 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:38.335433 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.335912 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.336373 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":175,"server-negotiator.queue_time_us":106}
I20260929 11:26:38.342507 25922 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:54070:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 11:26:38.343649 25922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:26:38.351665 26059 tablet_service.cc:1467] Processing CreateTablet for tablet 2694e3cdcc124092ac2311cb0038284b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a409e8aa760f4091a62718436a0f1bd5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:26:38.351854 26198 tablet_service.cc:1467] Processing CreateTablet for tablet 2694e3cdcc124092ac2311cb0038284b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a409e8aa760f4091a62718436a0f1bd5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:26:38.352857 26198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2694e3cdcc124092ac2311cb0038284b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:38.352861 26059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2694e3cdcc124092ac2311cb0038284b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:38.359231 26426 tablet_bootstrap.cc:492] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804: Bootstrap starting.
I20260929 11:26:38.358527 26337 tablet_service.cc:1467] Processing CreateTablet for tablet 2694e3cdcc124092ac2311cb0038284b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a409e8aa760f4091a62718436a0f1bd5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:26:38.359556 26427 tablet_bootstrap.cc:492] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Bootstrap starting.
I20260929 11:26:38.359616 26337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2694e3cdcc124092ac2311cb0038284b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:38.360919 26427 tablet_bootstrap.cc:654] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:38.361061 26426 tablet_bootstrap.cc:654] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:38.368561 26428 tablet_bootstrap.cc:492] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511: Bootstrap starting.
I20260929 11:26:38.370357 26428 tablet_bootstrap.cc:654] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:38.370658 26427 tablet_bootstrap.cc:492] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c: No bootstrap required, opened a new log
I20260929 11:26:38.370682 26426 tablet_bootstrap.cc:492] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804: No bootstrap required, opened a new log
I20260929 11:26:38.370818 26427 ts_tablet_manager.cc:1397] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260929 11:26:38.370831 26426 ts_tablet_manager.cc:1397] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20260929 11:26:38.371647 26426 raft_consensus.cc:348] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:38.371641 26427 raft_consensus.cc:348] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:38.371903 26426 raft_consensus.cc:374] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:38.371912 26427 raft_consensus.cc:374] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:38.371999 26426 raft_consensus.cc:729] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77cf22706b4f487c9d66a93f9bc02804, State: Initialized, Role: FOLLOWER
I20260929 11:26:38.371996 26427 raft_consensus.cc:729] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e46c2a7ac0341a4acd1de07bb4cbf4c, State: Initialized, Role: FOLLOWER
I20260929 11:26:38.372321 26427 consensus_queue.cc:260] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:38.372313 26426 consensus_queue.cc:260] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:38.372896 26427 ts_tablet_manager.cc:1428] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:26:38.373032 26428 tablet_bootstrap.cc:492] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511: No bootstrap required, opened a new log
I20260929 11:26:38.372812 26433 tablet_replica.cc:439] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } }
I20260929 11:26:38.373354 26428 ts_tablet_manager.cc:1397] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 11:26:38.373477 26433 tablet_replica.cc:442] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:26:38.374241 26428 raft_consensus.cc:348] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:38.374513 26428 raft_consensus.cc:374] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:38.374601 26428 raft_consensus.cc:729] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 390b6ee45e10487992567c61f5971511, State: Initialized, Role: FOLLOWER
I20260929 11:26:38.374703 26426 ts_tablet_manager.cc:1428] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260929 11:26:38.374614 26432 tablet_replica.cc:439] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } }
I20260929 11:26:38.375059 26432 tablet_replica.cc:442] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:26:38.375473 26428 consensus_queue.cc:260] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:38.376016 26434 tablet_replica.cc:439] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } }
I20260929 11:26:38.376287 26428 ts_tablet_manager.cc:1428] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 11:26:38.376416 26434 tablet_replica.cc:442] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:26:38.385748 26433 raft_consensus.cc:482] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:26:38.385995 26433 raft_consensus.cc:504] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:38.387697 26433 leader_election.cc:288] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77cf22706b4f487c9d66a93f9bc02804 (127.23.153.1:45795), 390b6ee45e10487992567c61f5971511 (127.23.153.3:33063)
I20260929 11:26:38.395547 26423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.388136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:46457 (local address 127.23.153.1:45795)
0929 11:26:38.388319 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.388325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.389047 (+ 722us) server_negotiation.cc:407] Connection header received
0929 11:26:38.389297 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.389303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.389409 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.389498 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:38.390574 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.391508 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.393552 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.394552 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.394691 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.394751 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.395149 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.395258 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:26:38.395803 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.388215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:45795 (local address 127.23.153.2:46457)
0929 11:26:38.388935 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:38.388949 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:38.389042 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:38.389559 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:38.389565 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:38.389605 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:38.390312 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.390336 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.391655 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.391661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.393325 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.393340 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.394772 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.394779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.394956 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.394994 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 11:26:38.395133 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 11:26:38.395279 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":56,"spinlock_wait_cycles":1280,"thread_start_us":427,"threads_started":1}
I20260929 11:26:38.396250 26422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.388190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:35271 (local address 127.23.153.3:33063)
0929 11:26:38.388366 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.388371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.388934 (+ 563us) server_negotiation.cc:407] Connection header received
0929 11:26:38.389294 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.389300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.389385 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.389498 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:38.390729 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.392049 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.394143 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.395242 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.395409 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.395482 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.395912 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.396067 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:26:38.396528 26079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2694e3cdcc124092ac2311cb0038284b" candidate_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77cf22706b4f487c9d66a93f9bc02804" is_pre_election: true
I20260929 11:26:38.396806 26079 raft_consensus.cc:2454] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e46c2a7ac0341a4acd1de07bb4cbf4c in term 0.
I20260929 11:26:38.396885 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.388030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:33063 (local address 127.23.153.2:35271)
0929 11:26:38.388810 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:38.388829 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:38.389004 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:38.389559 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:38.389565 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:38.389613 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:38.390370 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.390380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.392215 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.392220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.393928 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.393944 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.395513 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.395520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.395710 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.395747 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 11:26:38.396412 (+ 665us) client_negotiation.cc:240] Negotiation successful
0929 11:26:38.396587 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":29,"spinlock_wait_cycles":8704,"thread_start_us":355,"threads_started":1}
I20260929 11:26:38.397301 26150 leader_election.cc:302] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e46c2a7ac0341a4acd1de07bb4cbf4c, 77cf22706b4f487c9d66a93f9bc02804; no voters:
I20260929 11:26:38.397713 26433 raft_consensus.cc:2781] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:26:38.397825 26433 raft_consensus.cc:482] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:26:38.397917 26433 raft_consensus.cc:3037] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:38.397804 26357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2694e3cdcc124092ac2311cb0038284b" candidate_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "390b6ee45e10487992567c61f5971511" is_pre_election: true
I20260929 11:26:38.398224 26357 raft_consensus.cc:2454] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e46c2a7ac0341a4acd1de07bb4cbf4c in term 0.
I20260929 11:26:38.400250 26433 raft_consensus.cc:504] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:38.400949 26433 leader_election.cc:288] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [CANDIDATE]: Term 1 election: Requested vote from peers 77cf22706b4f487c9d66a93f9bc02804 (127.23.153.1:45795), 390b6ee45e10487992567c61f5971511 (127.23.153.3:33063)
I20260929 11:26:38.401479 26079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2694e3cdcc124092ac2311cb0038284b" candidate_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77cf22706b4f487c9d66a93f9bc02804"
I20260929 11:26:38.401517 26357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2694e3cdcc124092ac2311cb0038284b" candidate_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "390b6ee45e10487992567c61f5971511"
I20260929 11:26:38.401732 26079 raft_consensus.cc:3037] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:38.401764 26357 raft_consensus.cc:3037] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:38.404083 26357 raft_consensus.cc:2454] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e46c2a7ac0341a4acd1de07bb4cbf4c in term 1.
I20260929 11:26:38.404330 26079 raft_consensus.cc:2454] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e46c2a7ac0341a4acd1de07bb4cbf4c in term 1.
I20260929 11:26:38.404633 26152 leader_election.cc:302] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e46c2a7ac0341a4acd1de07bb4cbf4c, 390b6ee45e10487992567c61f5971511; no voters:
I20260929 11:26:38.405045 26433 raft_consensus.cc:2781] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:38.405575 26433 raft_consensus.cc:686] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 LEADER]: Becoming Leader. State: Replica: 2e46c2a7ac0341a4acd1de07bb4cbf4c, State: Running, Role: LEADER
I20260929 11:26:38.406132 26433 consensus_queue.cc:237] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:38.410943 26449 tablet_replica.cc:439] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c: TxnStatusTablet state changed. Reason: New leader 2e46c2a7ac0341a4acd1de07bb4cbf4c. Latest consensus state: current_term: 1 leader_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } }
I20260929 11:26:38.412379 26449 tablet_replica.cc:442] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:26:38.412724 25921 catalog_manager.cc:5495] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e46c2a7ac0341a4acd1de07bb4cbf4c (127.23.153.2). New cstate: current_term: 1 leader_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } health_report { overall_health: UNKNOWN } } }
I20260929 11:26:38.413318 26451 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 11:26:38.413671 26451 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 11:26:38.443046 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.435286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49086 (local address 127.23.153.2:45881)
0929 11:26:38.435450 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.435455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.435562 (+ 107us) server_negotiation.cc:407] Connection header received
0929 11:26:38.436079 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.436086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.436209 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.436421 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:38.436966 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.437889 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.439952 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.440440 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.440560 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.441323 (+ 763us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:38.442178 (+ 855us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:38.442373 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.442628 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.442731 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:26:38.443262 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.435181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:45881 (local address 127.0.0.1:49086)
0929 11:26:38.435460 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:38.435483 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:38.435660 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:38.436493 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:38.436498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:38.436515 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:26:38.436749 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.436760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.438083 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.438089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.439724 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.439746 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.440633 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.440641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.440867 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.440957 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:26:38.442363 (+ 1406us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:26:38.442404 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 11:26:38.442522 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 11:26:38.442714 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 11:26:38.453243 26357 raft_consensus.cc:1261] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511 [term 1 FOLLOWER]: Refusing update from remote peer 2e46c2a7ac0341a4acd1de07bb4cbf4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:38.453305 26079 raft_consensus.cc:1261] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804 [term 1 FOLLOWER]: Refusing update from remote peer 2e46c2a7ac0341a4acd1de07bb4cbf4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:38.454435 26433 consensus_queue.cc:1035] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:38.454761 26449 consensus_queue.cc:1035] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:38.466768 26432 tablet_replica.cc:439] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804: TxnStatusTablet state changed. Reason: New leader 2e46c2a7ac0341a4acd1de07bb4cbf4c. Latest consensus state: current_term: 1 leader_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } }
I20260929 11:26:38.467286 26432 tablet_replica.cc:442] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:26:38.470470 26454 mvcc.cc:205] Tried to move back new op lower bound from 7334630188852383744 to 7334630188675764224. Current Snapshot: MvccSnapshot[applied={T|T < 7334630188852383744}]
I20260929 11:26:38.470664 26434 tablet_replica.cc:439] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511: TxnStatusTablet state changed. Reason: New leader 2e46c2a7ac0341a4acd1de07bb4cbf4c. Latest consensus state: current_term: 1 leader_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } }
I20260929 11:26:38.471040 26434 tablet_replica.cc:442] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:26:38.475417 26432 tablet_replica.cc:439] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } }
I20260929 11:26:38.475701 26432 tablet_replica.cc:442] T 2694e3cdcc124092ac2311cb0038284b P 77cf22706b4f487c9d66a93f9bc02804: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 11:26:38.480443 26130 tablet.cc:2369] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:26:38.479736 26433 tablet_replica.cc:439] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } }
I20260929 11:26:38.481587 26433 tablet_replica.cc:442] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:26:38.481837 26433 tablet_replica.cc:439] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } }
I20260929 11:26:38.482033 26433 tablet_replica.cc:442] T 2694e3cdcc124092ac2311cb0038284b P 2e46c2a7ac0341a4acd1de07bb4cbf4c: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:26:38.483351 26456 mvcc.cc:205] Tried to move back new op lower bound from 7334630188852383744 to 7334630188675764224. Current Snapshot: MvccSnapshot[applied={T|T < 7334630188852383744}]
I20260929 11:26:38.487337 26434 tablet_replica.cc:439] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } } }
I20260929 11:26:38.487663 26434 tablet_replica.cc:442] T 2694e3cdcc124092ac2311cb0038284b P 390b6ee45e10487992567c61f5971511: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:26:38.497305 26422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.490663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38624 (local address 127.23.153.3:33063)
0929 11:26:38.490837 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.490843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.490886 (+ 43us) server_negotiation.cc:407] Connection header received
0929 11:26:38.490984 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.490990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.491138 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.491292 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:38.491728 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.492922 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.494403 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.495129 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.495269 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.495804 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:38.496502 (+ 698us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:38.496651 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.496779 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.496870 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:26:38.518640 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.512192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:49283 (local address 127.23.153.2:45881)
0929 11:26:38.512383 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.512388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.512567 (+ 179us) server_negotiation.cc:407] Connection header received
0929 11:26:38.512905 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.512911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.513031 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.513163 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:38.513671 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.514642 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.516707 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.517225 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.517336 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.517913 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:38.518032 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:38.518115 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.518287 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.518390 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:26:38.518755 26378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.512018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:45881 (local address 127.23.153.3:49283)
0929 11:26:38.512457 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:38.512484 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:38.512654 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:38.513209 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:38.513214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:38.513230 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:26:38.513448 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.513460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.514813 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.514819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.516516 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.516538 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.517396 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.517403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.517589 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.517680 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:26:38.518129 (+ 449us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:26:38.518141 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 11:26:38.518227 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 11:26:38.518425 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260929 11:26:38.533612 26079 raft_consensus.cc:1261] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804 [term 1 FOLLOWER]: Refusing update from remote peer 390b6ee45e10487992567c61f5971511: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:38.533828 26218 raft_consensus.cc:1261] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 FOLLOWER]: Refusing update from remote peer 390b6ee45e10487992567c61f5971511: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:38.534612 26436 consensus_queue.cc:1035] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:38.535358 26436 consensus_queue.cc:1035] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:38.605289 26422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.597663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:60197 (local address 127.23.153.3:33063)
0929 11:26:38.597890 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.597897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.598449 (+ 552us) server_negotiation.cc:407] Connection header received
0929 11:26:38.598623 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.598629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.598726 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.598840 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:38.599570 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.600627 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.602535 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.603266 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.603429 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.604426 (+ 997us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:38.604618 (+ 192us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:38.604707 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.604894 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.604993 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:26:38.605654 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.597672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:33063 (local address 127.23.153.2:60197)
0929 11:26:38.598341 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:38.598359 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:38.598504 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:38.598904 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:38.598910 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:38.598925 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:26:38.599379 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.599397 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.600768 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.600773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.602363 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.602378 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.603655 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.603667 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.603951 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.604077 (+ 126us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:26:38.604757 (+ 680us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:26:38.604774 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 11:26:38.604863 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 11:26:38.605080 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":301,"threads_started":1}
I20260929 11:26:38.654453 25922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:26:38.654764 25922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:26:38.659421 25922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb: Sending DeleteTablet for 3 replicas of tablet 39176ed2815c46759ba81f19be330353
I20260929 11:26:38.660986 26337 tablet_service.cc:1514] Processing DeleteTablet for tablet 39176ed2815c46759ba81f19be330353 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:26:38 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:38620
I20260929 11:26:38.661219 26059 tablet_service.cc:1514] Processing DeleteTablet for tablet 39176ed2815c46759ba81f19be330353 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:26:38 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:56608
I20260929 11:26:38.662263 26198 tablet_service.cc:1514] Processing DeleteTablet for tablet 39176ed2815c46759ba81f19be330353 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:26:38 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:49070
I20260929 11:26:38.664711 25922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:26:38.665462 25922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:26:38.671542 26474 tablet_replica.cc:330] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511: stopping tablet replica
I20260929 11:26:38.672137 26474 raft_consensus.cc:2229] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:26:38.672847 26474 raft_consensus.cc:2258] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:26:38.679318 26476 tablet_replica.cc:330] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: stopping tablet replica
I20260929 11:26:38.680675 26058 tablet_service.cc:1467] Processing CreateTablet for tablet e59456965f364cf9856b522ab9598ef7 (DEFAULT_TABLE table=test-table [id=dd7f03868df546898818973a43884cc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:38.681277 26058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e59456965f364cf9856b522ab9598ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:38.681839 26476 raft_consensus.cc:2229] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:26:38.682099 26475 tablet_replica.cc:330] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804: stopping tablet replica
I20260929 11:26:38.682569 26475 raft_consensus.cc:2229] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:26:38.684402 26476 raft_consensus.cc:2258] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:26:38.684837 26475 raft_consensus.cc:2258] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:26:38.686477 26474 ts_tablet_manager.cc:1905] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:26:38.686826 26337 tablet_service.cc:1467] Processing CreateTablet for tablet e59456965f364cf9856b522ab9598ef7 (DEFAULT_TABLE table=test-table [id=dd7f03868df546898818973a43884cc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:38.687726 26337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e59456965f364cf9856b522ab9598ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:38.688875 26476 ts_tablet_manager.cc:1905] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:26:38.691609 26197 tablet_service.cc:1467] Processing CreateTablet for tablet e59456965f364cf9856b522ab9598ef7 (DEFAULT_TABLE table=test-table [id=dd7f03868df546898818973a43884cc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:38.692217 26197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e59456965f364cf9856b522ab9598ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:38.694254 26428 tablet_bootstrap.cc:492] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511: Bootstrap starting.
I20260929 11:26:38.695996 26428 tablet_bootstrap.cc:654] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:38.698856 26426 tablet_bootstrap.cc:492] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804: Bootstrap starting.
I20260929 11:26:38.700821 26426 tablet_bootstrap.cc:654] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:38.702551 26476 ts_tablet_manager.cc:1918] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:26:38.702876 26476 log.cc:1196] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/39176ed2815c46759ba81f19be330353
I20260929 11:26:38.703639 26476 ts_tablet_manager.cc:1939] T 39176ed2815c46759ba81f19be330353 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Deleting consensus metadata
I20260929 11:26:38.705808 25909 catalog_manager.cc:4841] TS 2e46c2a7ac0341a4acd1de07bb4cbf4c (127.23.153.2:45881): tablet 39176ed2815c46759ba81f19be330353 (table test-table [id=18c86896990b4cf7b530059e37e03715]) successfully deleted
I20260929 11:26:38.709180 26475 ts_tablet_manager.cc:1905] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:26:38.713099 26474 ts_tablet_manager.cc:1918] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:26:38.713383 26474 log.cc:1196] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/39176ed2815c46759ba81f19be330353
I20260929 11:26:38.713976 26474 ts_tablet_manager.cc:1939] T 39176ed2815c46759ba81f19be330353 P 390b6ee45e10487992567c61f5971511: Deleting consensus metadata
I20260929 11:26:38.717870 26427 tablet_bootstrap.cc:492] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Bootstrap starting.
I20260929 11:26:38.719818 26475 ts_tablet_manager.cc:1918] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:26:38.720131 26475 log.cc:1196] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/39176ed2815c46759ba81f19be330353
I20260929 11:26:38.720129 26427 tablet_bootstrap.cc:654] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:38.720763 26475 ts_tablet_manager.cc:1939] T 39176ed2815c46759ba81f19be330353 P 77cf22706b4f487c9d66a93f9bc02804: Deleting consensus metadata
I20260929 11:26:38.723030 25907 catalog_manager.cc:4841] TS 77cf22706b4f487c9d66a93f9bc02804 (127.23.153.1:45795): tablet 39176ed2815c46759ba81f19be330353 (table test-table [id=18c86896990b4cf7b530059e37e03715]) successfully deleted
I20260929 11:26:38.733206 25909 catalog_manager.cc:4841] TS 390b6ee45e10487992567c61f5971511 (127.23.153.3:33063): tablet 39176ed2815c46759ba81f19be330353 (table test-table [id=18c86896990b4cf7b530059e37e03715]) successfully deleted
I20260929 11:26:38.732666 26426 tablet_bootstrap.cc:492] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804: No bootstrap required, opened a new log
I20260929 11:26:38.733688 26426 ts_tablet_manager.cc:1397] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.024s
I20260929 11:26:38.734562 26426 raft_consensus.cc:348] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:38.734822 26426 raft_consensus.cc:374] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:38.734897 26426 raft_consensus.cc:729] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77cf22706b4f487c9d66a93f9bc02804, State: Initialized, Role: FOLLOWER
I20260929 11:26:38.735258 26426 consensus_queue.cc:260] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:38.736604 26428 tablet_bootstrap.cc:492] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511: No bootstrap required, opened a new log
I20260929 11:26:38.736604 26426 ts_tablet_manager.cc:1428] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 11:26:38.736785 26428 ts_tablet_manager.cc:1397] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.011s
I20260929 11:26:38.737437 26428 raft_consensus.cc:348] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:38.737656 26428 raft_consensus.cc:374] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:38.737732 26428 raft_consensus.cc:729] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 390b6ee45e10487992567c61f5971511, State: Initialized, Role: FOLLOWER
I20260929 11:26:38.737969 26428 consensus_queue.cc:260] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:38.739138 26427 tablet_bootstrap.cc:492] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: No bootstrap required, opened a new log
I20260929 11:26:38.739388 26427 ts_tablet_manager.cc:1397] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.000s
I20260929 11:26:38.740051 26428 ts_tablet_manager.cc:1428] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 11:26:38.740350 26427 raft_consensus.cc:348] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:38.740687 26427 raft_consensus.cc:374] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:38.740811 26427 raft_consensus.cc:729] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e46c2a7ac0341a4acd1de07bb4cbf4c, State: Initialized, Role: FOLLOWER
I20260929 11:26:38.741169 26427 consensus_queue.cc:260] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:38.741856 26427 ts_tablet_manager.cc:1428] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:26:38.867866 26432 raft_consensus.cc:482] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:26:38.868172 26432 raft_consensus.cc:504] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:38.870257 26432 leader_election.cc:288] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e46c2a7ac0341a4acd1de07bb4cbf4c (127.23.153.2:45881), 390b6ee45e10487992567c61f5971511 (127.23.153.3:33063)
I20260929 11:26:38.880290 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.871001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:52051 (local address 127.23.153.2:45881)
0929 11:26:38.871263 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.871270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.871769 (+ 499us) server_negotiation.cc:407] Connection header received
0929 11:26:38.872097 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.872104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.872190 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.872257 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:38.873032 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.874157 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.877409 (+ 3252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.878829 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.879007 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.879144 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.879819 (+ 675us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.879979 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 11:26:38.881397 26480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.870871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:45881 (local address 127.23.153.1:52051)
0929 11:26:38.871645 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:38.871666 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:38.871902 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:38.872457 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:38.872461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:38.872489 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:38.872835 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.872847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.874383 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.874390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.877113 (+ 2723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.877141 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.879389 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.879396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.879607 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.879643 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 11:26:38.880830 (+ 1187us) client_negotiation.cc:240] Negotiation successful
0929 11:26:38.881010 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":504,"threads_started":1}
I20260929 11:26:38.885250 26218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59456965f364cf9856b522ab9598ef7" candidate_uuid: "77cf22706b4f487c9d66a93f9bc02804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" is_pre_election: true
I20260929 11:26:38.885615 26218 raft_consensus.cc:2454] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77cf22706b4f487c9d66a93f9bc02804 in term 0.
I20260929 11:26:38.886353 26012 leader_election.cc:302] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e46c2a7ac0341a4acd1de07bb4cbf4c, 77cf22706b4f487c9d66a93f9bc02804; no voters:
I20260929 11:26:38.886857 26432 raft_consensus.cc:2781] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:26:38.886978 26432 raft_consensus.cc:482] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:26:38.887094 26432 raft_consensus.cc:3037] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:38.890640 26422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.872349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:55511 (local address 127.23.153.3:33063)
0929 11:26:38.872567 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.872574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.882016 (+ 9442us) server_negotiation.cc:407] Connection header received
0929 11:26:38.882326 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.882332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.882422 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.882485 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:26:38.883360 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.884662 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.887829 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.889275 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.889455 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.889540 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.890154 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.890323 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:26:38.891152 26432 raft_consensus.cc:504] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:38.891841 26480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.871876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:33063 (local address 127.23.153.1:55511)
0929 11:26:38.881890 (+ 10014us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:38.881914 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:38.882162 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:38.882693 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:38.882698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:38.882726 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:26:38.882914 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.882924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.884878 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.884885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.887556 (+ 2671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.887577 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.889718 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.889726 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.889945 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.889982 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 11:26:38.891278 (+ 1296us) client_negotiation.cc:240] Negotiation successful
0929 11:26:38.891527 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9863,"thread_start_us":12374,"threads_started":1}
I20260929 11:26:38.891924 26432 leader_election.cc:288] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [CANDIDATE]: Term 1 election: Requested vote from peers 2e46c2a7ac0341a4acd1de07bb4cbf4c (127.23.153.2:45881), 390b6ee45e10487992567c61f5971511 (127.23.153.3:33063)
I20260929 11:26:38.892796 26218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59456965f364cf9856b522ab9598ef7" candidate_uuid: "77cf22706b4f487c9d66a93f9bc02804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c"
I20260929 11:26:38.893066 26218 raft_consensus.cc:3037] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:38.893196 26357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59456965f364cf9856b522ab9598ef7" candidate_uuid: "77cf22706b4f487c9d66a93f9bc02804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "390b6ee45e10487992567c61f5971511" is_pre_election: true
I20260929 11:26:38.893460 26357 raft_consensus.cc:2454] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77cf22706b4f487c9d66a93f9bc02804 in term 0.
I20260929 11:26:38.894236 26357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e59456965f364cf9856b522ab9598ef7" candidate_uuid: "77cf22706b4f487c9d66a93f9bc02804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "390b6ee45e10487992567c61f5971511"
I20260929 11:26:38.894459 26357 raft_consensus.cc:3037] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:38.896610 26218 raft_consensus.cc:2454] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77cf22706b4f487c9d66a93f9bc02804 in term 1.
I20260929 11:26:38.897526 26012 leader_election.cc:302] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e46c2a7ac0341a4acd1de07bb4cbf4c, 77cf22706b4f487c9d66a93f9bc02804; no voters:
I20260929 11:26:38.897917 26432 raft_consensus.cc:2781] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:38.898144 26357 raft_consensus.cc:2454] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77cf22706b4f487c9d66a93f9bc02804 in term 1.
I20260929 11:26:38.898676 26432 raft_consensus.cc:686] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [term 1 LEADER]: Becoming Leader. State: Replica: 77cf22706b4f487c9d66a93f9bc02804, State: Running, Role: LEADER
I20260929 11:26:38.899207 26432 consensus_queue.cc:237] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:38.903650 25921 catalog_manager.cc:5495] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77cf22706b4f487c9d66a93f9bc02804 (127.23.153.1). New cstate: current_term: 1 leader_uuid: "77cf22706b4f487c9d66a93f9bc02804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } health_report { overall_health: UNKNOWN } } }
I20260929 11:26:38.946883 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.939163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56620 (local address 127.23.153.1:45795)
0929 11:26:38.940042 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.940048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.940071 (+ 23us) server_negotiation.cc:407] Connection header received
0929 11:26:38.940175 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.940180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.940274 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.940453 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:38.940843 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.941945 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.943769 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.944439 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.944631 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.945279 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:38.946067 (+ 788us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:38.946242 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.946386 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.946483 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":584,"threads_started":1}
I20260929 11:26:38.966089 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.959111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:37887 (local address 127.23.153.2:45881)
0929 11:26:38.959295 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:38.959302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:38.959923 (+ 621us) server_negotiation.cc:407] Connection header received
0929 11:26:38.960177 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:38.960181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:38.960255 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:38.960346 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:38.960915 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.961898 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.964198 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.964657 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.964787 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.965416 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:38.965542 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:38.965618 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:38.965757 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 11:26:38.965830 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 11:26:38.966290 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:38.958934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:45881 (local address 127.23.153.1:37887)
0929 11:26:38.959793 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:38.959813 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:38.960000 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:38.960381 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:38.960386 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:38.960401 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:26:38.960758 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.960771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.962120 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.962127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.963878 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:38.963905 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:38.964899 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:38.964906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:38.965106 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:38.965196 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:26:38.965646 (+ 450us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:26:38.965657 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 11:26:38.965738 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 11:26:38.965909 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":466,"threads_started":1}
I20260929 11:26:38.975924 26357 raft_consensus.cc:1261] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511 [term 1 FOLLOWER]: Refusing update from remote peer 77cf22706b4f487c9d66a93f9bc02804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:38.976197 26218 raft_consensus.cc:1261] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 FOLLOWER]: Refusing update from remote peer 77cf22706b4f487c9d66a93f9bc02804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:38.976933 26482 consensus_queue.cc:1035] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:38.977751 26432 consensus_queue.cc:1035] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:38.990586 26455 mvcc.cc:205] Tried to move back new op lower bound from 7334630190999121920 to 7334630190696484864. Current Snapshot: MvccSnapshot[applied={T|T < 7334630190999121920}]
I20260929 11:26:39.020829 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:39.014337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:45795 (local address 127.23.153.2:50697)
0929 11:26:39.014568 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:39.014589 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:39.014713 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:39.015283 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:39.015287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:39.015301 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:26:39.015534 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:39.015546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:39.016956 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:39.016962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:39.018159 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:39.018177 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:39.019551 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:39.019557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:39.019755 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:39.019811 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:26:39.020290 (+ 479us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:26:39.020301 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 11:26:39.020371 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 11:26:39.020528 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 11:26:39.021409 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:39.014749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:50697 (local address 127.23.153.1:45795)
0929 11:26:39.014942 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:39.014948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:39.014967 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:26:39.015054 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:39.015059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:39.015150 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:39.015242 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:39.015713 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:39.016733 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:39.018381 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:39.019001 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:39.019155 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:39.020024 (+ 869us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:39.020164 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:39.020241 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:39.021007 (+ 766us) server_negotiation.cc:299] Negotiation successful
0929 11:26:39.021116 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 11:26:39.022483 26455 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 11:26:40.637291 25921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:26:40.637552 25921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:26:40.642632 25921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb: Sending DeleteTablet for 3 replicas of tablet e59456965f364cf9856b522ab9598ef7
I20260929 11:26:40.643843 26337 tablet_service.cc:1514] Processing DeleteTablet for tablet e59456965f364cf9856b522ab9598ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:26:40 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:38620
I20260929 11:26:40.643865 26059 tablet_service.cc:1514] Processing DeleteTablet for tablet e59456965f364cf9856b522ab9598ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:26:40 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:56608
I20260929 11:26:40.643815 26197 tablet_service.cc:1514] Processing DeleteTablet for tablet e59456965f364cf9856b522ab9598ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:26:40 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:49070
I20260929 11:26:40.655917 26513 tablet_replica.cc:330] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: stopping tablet replica
I20260929 11:26:40.656683 26514 tablet_replica.cc:330] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804: stopping tablet replica
I20260929 11:26:40.657069 26514 raft_consensus.cc:2229] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:26:40.656175 26513 raft_consensus.cc:2229] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:26:40.657696 26513 raft_consensus.cc:2258] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:26:40.658300 26514 raft_consensus.cc:2258] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:26:40.659003 26513 ts_tablet_manager.cc:1905] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:26:40.659391 26514 ts_tablet_manager.cc:1905] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:26:40.663630 26512 tablet_replica.cc:330] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511: stopping tablet replica
I20260929 11:26:40.663866 26512 raft_consensus.cc:2229] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:26:40.664129 26512 raft_consensus.cc:2258] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:26:40.665437 26512 ts_tablet_manager.cc:1905] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:26:40.667402 26514 ts_tablet_manager.cc:1918] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 11:26:40.667531 26513 ts_tablet_manager.cc:1918] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 11:26:40.667626 26514 log.cc:1196] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/e59456965f364cf9856b522ab9598ef7
I20260929 11:26:40.667745 26513 log.cc:1196] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/e59456965f364cf9856b522ab9598ef7
I20260929 11:26:40.668156 26514 ts_tablet_manager.cc:1939] T e59456965f364cf9856b522ab9598ef7 P 77cf22706b4f487c9d66a93f9bc02804: Deleting consensus metadata
I20260929 11:26:40.668349 26513 ts_tablet_manager.cc:1939] T e59456965f364cf9856b522ab9598ef7 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Deleting consensus metadata
I20260929 11:26:40.669551 25907 catalog_manager.cc:4841] TS 77cf22706b4f487c9d66a93f9bc02804 (127.23.153.1:45795): tablet e59456965f364cf9856b522ab9598ef7 (table test-table [id=dd7f03868df546898818973a43884cc0]) successfully deleted
I20260929 11:26:40.669561 25909 catalog_manager.cc:4841] TS 2e46c2a7ac0341a4acd1de07bb4cbf4c (127.23.153.2:45881): tablet e59456965f364cf9856b522ab9598ef7 (table test-table [id=dd7f03868df546898818973a43884cc0]) successfully deleted
I20260929 11:26:40.671679 26512 ts_tablet_manager.cc:1918] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 11:26:40.671801 26512 log.cc:1196] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/e59456965f364cf9856b522ab9598ef7
I20260929 11:26:40.672210 26512 ts_tablet_manager.cc:1939] T e59456965f364cf9856b522ab9598ef7 P 390b6ee45e10487992567c61f5971511: Deleting consensus metadata
I20260929 11:26:40.673445 25909 catalog_manager.cc:4841] TS 390b6ee45e10487992567c61f5971511 (127.23.153.3:33063): tablet e59456965f364cf9856b522ab9598ef7 (table test-table [id=dd7f03868df546898818973a43884cc0]) successfully deleted
Sep 29 11:26:40 dist-test-slave-qk05 krb5kdc[25877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681200, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 11:26:40 dist-test-slave-qk05 krb5kdc[25877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681200, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:40.710702 26522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:40.690399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54072 (local address 127.23.153.62:38829)
0929 11:26:40.691084 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:40.691090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:40.691118 (+ 28us) server_negotiation.cc:407] Connection header received
0929 11:26:40.691239 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:40.691243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:40.691310 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:40.691407 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:40.692791 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:40.693585 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:40.695244 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:40.695744 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:40.695868 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:40.705409 (+ 9541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:40.705416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:40.705420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:40.705464 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:40.708134 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:40.708941 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:40.708944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:40.708947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:40.709072 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:40.709351 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:40.709354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:40.709356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:40.709635 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:40.709812 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:40.710186 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 11:26:40.710364 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":311,"threads_started":1}
I20260929 11:26:40.714331 25921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:26:40.714955 25921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:26:40.721975 26059 tablet_service.cc:1467] Processing CreateTablet for tablet 437dc2eb1b414ebd9899e72903e18088 (DEFAULT_TABLE table=test-table [id=5291c83145734a23a5c1afa29c4551ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:40.722039 26197 tablet_service.cc:1467] Processing CreateTablet for tablet 437dc2eb1b414ebd9899e72903e18088 (DEFAULT_TABLE table=test-table [id=5291c83145734a23a5c1afa29c4551ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:40.722348 26337 tablet_service.cc:1467] Processing CreateTablet for tablet 437dc2eb1b414ebd9899e72903e18088 (DEFAULT_TABLE table=test-table [id=5291c83145734a23a5c1afa29c4551ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:40.722468 26059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437dc2eb1b414ebd9899e72903e18088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:40.722483 26197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437dc2eb1b414ebd9899e72903e18088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:40.722721 26337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437dc2eb1b414ebd9899e72903e18088. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:40.731078 26523 tablet_bootstrap.cc:492] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Bootstrap starting.
I20260929 11:26:40.732604 26523 tablet_bootstrap.cc:654] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:40.734406 26525 tablet_bootstrap.cc:492] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804: Bootstrap starting.
I20260929 11:26:40.736443 26525 tablet_bootstrap.cc:654] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:40.738481 26524 tablet_bootstrap.cc:492] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511: Bootstrap starting.
I20260929 11:26:40.740020 26523 tablet_bootstrap.cc:492] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: No bootstrap required, opened a new log
I20260929 11:26:40.740235 26523 ts_tablet_manager.cc:1397] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260929 11:26:40.740264 26524 tablet_bootstrap.cc:654] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:40.741122 26523 raft_consensus.cc:348] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:40.741436 26523 raft_consensus.cc:374] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:40.741559 26523 raft_consensus.cc:729] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e46c2a7ac0341a4acd1de07bb4cbf4c, State: Initialized, Role: FOLLOWER
I20260929 11:26:40.741956 26523 consensus_queue.cc:260] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:40.742611 26523 ts_tablet_manager.cc:1428] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 11:26:40.742741 26525 tablet_bootstrap.cc:492] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804: No bootstrap required, opened a new log
I20260929 11:26:40.743001 26525 ts_tablet_manager.cc:1397] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260929 11:26:40.743396 26524 tablet_bootstrap.cc:492] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511: No bootstrap required, opened a new log
I20260929 11:26:40.743548 26524 ts_tablet_manager.cc:1397] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 11:26:40.743860 26525 raft_consensus.cc:348] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:40.744400 26525 raft_consensus.cc:374] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:40.744308 26524 raft_consensus.cc:348] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:40.744546 26525 raft_consensus.cc:729] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77cf22706b4f487c9d66a93f9bc02804, State: Initialized, Role: FOLLOWER
I20260929 11:26:40.744635 26524 raft_consensus.cc:374] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:40.744773 26524 raft_consensus.cc:729] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 390b6ee45e10487992567c61f5971511, State: Initialized, Role: FOLLOWER
I20260929 11:26:40.744879 26525 consensus_queue.cc:260] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:40.745049 26524 consensus_queue.cc:260] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:40.745436 26525 ts_tablet_manager.cc:1428] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 11:26:40.745886 26524 ts_tablet_manager.cc:1428] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 11:26:40.899467 26494 raft_consensus.cc:482] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:26:40.899680 26494 raft_consensus.cc:504] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:40.900334 26494 leader_election.cc:288] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77cf22706b4f487c9d66a93f9bc02804 (127.23.153.1:45795), 390b6ee45e10487992567c61f5971511 (127.23.153.3:33063)
I20260929 11:26:40.901203 26079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437dc2eb1b414ebd9899e72903e18088" candidate_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77cf22706b4f487c9d66a93f9bc02804" is_pre_election: true
I20260929 11:26:40.901202 26356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437dc2eb1b414ebd9899e72903e18088" candidate_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "390b6ee45e10487992567c61f5971511" is_pre_election: true
I20260929 11:26:40.901494 26079 raft_consensus.cc:2454] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e46c2a7ac0341a4acd1de07bb4cbf4c in term 0.
I20260929 11:26:40.901497 26356 raft_consensus.cc:2454] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e46c2a7ac0341a4acd1de07bb4cbf4c in term 0.
I20260929 11:26:40.902045 26152 leader_election.cc:302] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2e46c2a7ac0341a4acd1de07bb4cbf4c, 390b6ee45e10487992567c61f5971511, 77cf22706b4f487c9d66a93f9bc02804; no voters:
I20260929 11:26:40.902364 26494 raft_consensus.cc:2781] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:26:40.902522 26494 raft_consensus.cc:482] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:26:40.902604 26494 raft_consensus.cc:3037] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:40.905136 26494 raft_consensus.cc:504] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:40.905817 26494 leader_election.cc:288] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [CANDIDATE]: Term 1 election: Requested vote from peers 77cf22706b4f487c9d66a93f9bc02804 (127.23.153.1:45795), 390b6ee45e10487992567c61f5971511 (127.23.153.3:33063)
I20260929 11:26:40.906476 26356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437dc2eb1b414ebd9899e72903e18088" candidate_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "390b6ee45e10487992567c61f5971511"
I20260929 11:26:40.906476 26079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "437dc2eb1b414ebd9899e72903e18088" candidate_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77cf22706b4f487c9d66a93f9bc02804"
I20260929 11:26:40.906682 26356 raft_consensus.cc:3037] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:40.906677 26079 raft_consensus.cc:3037] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:40.908823 26356 raft_consensus.cc:2454] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e46c2a7ac0341a4acd1de07bb4cbf4c in term 1.
I20260929 11:26:40.908854 26079 raft_consensus.cc:2454] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e46c2a7ac0341a4acd1de07bb4cbf4c in term 1.
I20260929 11:26:40.909403 26152 leader_election.cc:302] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2e46c2a7ac0341a4acd1de07bb4cbf4c, 390b6ee45e10487992567c61f5971511, 77cf22706b4f487c9d66a93f9bc02804; no voters:
I20260929 11:26:40.909716 26494 raft_consensus.cc:2781] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:40.909866 26494 raft_consensus.cc:686] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 LEADER]: Becoming Leader. State: Replica: 2e46c2a7ac0341a4acd1de07bb4cbf4c, State: Running, Role: LEADER
I20260929 11:26:40.910144 26494 consensus_queue.cc:237] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:40.912892 25921 catalog_manager.cc:5495] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e46c2a7ac0341a4acd1de07bb4cbf4c (127.23.153.2). New cstate: current_term: 1 leader_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } health_report { overall_health: UNKNOWN } } }
I20260929 11:26:40.950608 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:40.945458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49102 (local address 127.23.153.2:45881)
0929 11:26:40.946099 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:40.946104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:40.946126 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:26:40.946185 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:40.946188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:40.946254 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:40.946352 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:40.946784 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:40.947580 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:40.948904 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:40.949305 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:40.949420 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:40.949850 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:40.949974 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:40.950051 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:40.950200 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 11:26:40.950279 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":311,"threads_started":1}
I20260929 11:26:40.967078 26535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:40.961555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:45881 (local address 127.23.153.2:43571)
0929 11:26:40.962430 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:40.962449 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:40.962608 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:40.962963 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:40.962969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:40.962983 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:26:40.963242 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:40.963251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:40.964250 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:40.964254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:40.965281 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:40.965293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:40.965973 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:40.965980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:40.966097 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:40.966132 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:26:40.966460 (+ 328us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:26:40.966469 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 11:26:40.966536 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 11:26:40.966697 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":1,"thread_start_us":419,"threads_started":1}
I20260929 11:26:40.967078 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:40.961818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:43571 (local address 127.23.153.2:45881)
0929 11:26:40.962166 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:40.962171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:40.962544 (+ 373us) server_negotiation.cc:407] Connection header received
0929 11:26:40.962723 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:40.962728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:40.962793 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:40.962879 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:40.963386 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:40.964117 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:40.965427 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:40.965811 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:40.965925 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:40.966262 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:40.966368 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:40.966449 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:40.966558 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 11:26:40.966659 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":147}
I20260929 11:26:40.977171 26079 raft_consensus.cc:1261] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804 [term 1 FOLLOWER]: Refusing update from remote peer 2e46c2a7ac0341a4acd1de07bb4cbf4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:40.977146 26356 raft_consensus.cc:1261] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511 [term 1 FOLLOWER]: Refusing update from remote peer 2e46c2a7ac0341a4acd1de07bb4cbf4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:40.977843 26499 consensus_queue.cc:1035] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:40.978457 26499 consensus_queue.cc:1035] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:41.035310 25920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:26:41.035557 25920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:26:41.039063 25920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb: Sending DeleteTablet for 3 replicas of tablet 437dc2eb1b414ebd9899e72903e18088
I20260929 11:26:41.040223 26059 tablet_service.cc:1514] Processing DeleteTablet for tablet 437dc2eb1b414ebd9899e72903e18088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:26:41 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:56608
I20260929 11:26:41.040457 26197 tablet_service.cc:1514] Processing DeleteTablet for tablet 437dc2eb1b414ebd9899e72903e18088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:26:41 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:49070
I20260929 11:26:41.040571 26514 tablet_replica.cc:330] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804: stopping tablet replica
I20260929 11:26:41.040795 26513 tablet_replica.cc:330] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: stopping tablet replica
I20260929 11:26:41.040807 26514 raft_consensus.cc:2229] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:26:41.040818 26337 tablet_service.cc:1514] Processing DeleteTablet for tablet 437dc2eb1b414ebd9899e72903e18088 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:26:41 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:38620
I20260929 11:26:41.041023 26514 raft_consensus.cc:2258] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:26:41.041113 26512 tablet_replica.cc:330] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511: stopping tablet replica
I20260929 11:26:41.041296 26512 raft_consensus.cc:2229] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:26:41.041328 26513 raft_consensus.cc:2229] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:26:41.041495 26512 raft_consensus.cc:2258] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:26:41.041887 26513 raft_consensus.cc:2258] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:26:41.042291 26514 ts_tablet_manager.cc:1905] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:26:41.042783 26512 ts_tablet_manager.cc:1905] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:26:41.043339 26513 ts_tablet_manager.cc:1905] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:26:41.049539 26512 ts_tablet_manager.cc:1918] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:26:41.049707 26512 log.cc:1196] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/437dc2eb1b414ebd9899e72903e18088
I20260929 11:26:41.049660 26514 ts_tablet_manager.cc:1918] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:26:41.049834 26514 log.cc:1196] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/437dc2eb1b414ebd9899e72903e18088
I20260929 11:26:41.050107 26512 ts_tablet_manager.cc:1939] T 437dc2eb1b414ebd9899e72903e18088 P 390b6ee45e10487992567c61f5971511: Deleting consensus metadata
I20260929 11:26:41.050302 26514 ts_tablet_manager.cc:1939] T 437dc2eb1b414ebd9899e72903e18088 P 77cf22706b4f487c9d66a93f9bc02804: Deleting consensus metadata
I20260929 11:26:41.050289 26513 ts_tablet_manager.cc:1918] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:26:41.050437 26513 log.cc:1196] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/437dc2eb1b414ebd9899e72903e18088
I20260929 11:26:41.050846 26513 ts_tablet_manager.cc:1939] T 437dc2eb1b414ebd9899e72903e18088 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Deleting consensus metadata
I20260929 11:26:41.051383 25909 catalog_manager.cc:4841] TS 390b6ee45e10487992567c61f5971511 (127.23.153.3:33063): tablet 437dc2eb1b414ebd9899e72903e18088 (table test-table [id=5291c83145734a23a5c1afa29c4551ad]) successfully deleted
I20260929 11:26:41.051894 25920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:26:41.052594 25920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:26:41.052788 25909 catalog_manager.cc:4841] TS 2e46c2a7ac0341a4acd1de07bb4cbf4c (127.23.153.2:45881): tablet 437dc2eb1b414ebd9899e72903e18088 (table test-table [id=5291c83145734a23a5c1afa29c4551ad]) successfully deleted
I20260929 11:26:41.052856 25907 catalog_manager.cc:4841] TS 77cf22706b4f487c9d66a93f9bc02804 (127.23.153.1:45795): tablet 437dc2eb1b414ebd9899e72903e18088 (table test-table [id=5291c83145734a23a5c1afa29c4551ad]) successfully deleted
I20260929 11:26:41.060289 26059 tablet_service.cc:1467] Processing CreateTablet for tablet a778cb2abdee40b1b29b0b330698d046 (DEFAULT_TABLE table=test-table [id=f7dcd0b55f5c4b5ba03221d572044654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:41.060632 26337 tablet_service.cc:1467] Processing CreateTablet for tablet a778cb2abdee40b1b29b0b330698d046 (DEFAULT_TABLE table=test-table [id=f7dcd0b55f5c4b5ba03221d572044654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:41.060818 26059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a778cb2abdee40b1b29b0b330698d046. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:41.061028 26337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a778cb2abdee40b1b29b0b330698d046. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:41.063670 26197 tablet_service.cc:1467] Processing CreateTablet for tablet a778cb2abdee40b1b29b0b330698d046 (DEFAULT_TABLE table=test-table [id=f7dcd0b55f5c4b5ba03221d572044654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:26:41.064054 26197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a778cb2abdee40b1b29b0b330698d046. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:41.067795 26525 tablet_bootstrap.cc:492] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804: Bootstrap starting.
I20260929 11:26:41.069517 26525 tablet_bootstrap.cc:654] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:41.070904 26523 tablet_bootstrap.cc:492] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Bootstrap starting.
I20260929 11:26:41.072178 26525 tablet_bootstrap.cc:492] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804: No bootstrap required, opened a new log
I20260929 11:26:41.072453 26525 ts_tablet_manager.cc:1397] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260929 11:26:41.072705 26523 tablet_bootstrap.cc:654] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:41.073273 26525 raft_consensus.cc:348] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:41.073535 26525 raft_consensus.cc:374] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:41.073609 26525 raft_consensus.cc:729] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77cf22706b4f487c9d66a93f9bc02804, State: Initialized, Role: FOLLOWER
I20260929 11:26:41.073892 26525 consensus_queue.cc:260] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:41.077535 26525 ts_tablet_manager.cc:1428] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260929 11:26:41.078111 26524 tablet_bootstrap.cc:492] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511: Bootstrap starting.
I20260929 11:26:41.080178 26524 tablet_bootstrap.cc:654] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:41.083196 26524 tablet_bootstrap.cc:492] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511: No bootstrap required, opened a new log
I20260929 11:26:41.083483 26524 ts_tablet_manager.cc:1397] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260929 11:26:41.084318 26524 raft_consensus.cc:348] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:41.084657 26524 raft_consensus.cc:374] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:41.084770 26524 raft_consensus.cc:729] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 390b6ee45e10487992567c61f5971511, State: Initialized, Role: FOLLOWER
I20260929 11:26:41.085130 26524 consensus_queue.cc:260] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:41.086477 26524 ts_tablet_manager.cc:1428] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 11:26:41.089646 26523 tablet_bootstrap.cc:492] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: No bootstrap required, opened a new log
I20260929 11:26:41.089861 26523 ts_tablet_manager.cc:1397] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.002s
I20260929 11:26:41.090713 26523 raft_consensus.cc:348] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:41.091035 26523 raft_consensus.cc:374] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:41.091137 26523 raft_consensus.cc:729] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e46c2a7ac0341a4acd1de07bb4cbf4c, State: Initialized, Role: FOLLOWER
I20260929 11:26:41.091432 26523 consensus_queue.cc:260] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:41.092046 26523 ts_tablet_manager.cc:1428] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 11:26:41.197374 26504 raft_consensus.cc:482] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:26:41.197597 26504 raft_consensus.cc:504] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:41.198276 26504 leader_election.cc:288] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e46c2a7ac0341a4acd1de07bb4cbf4c (127.23.153.2:45881), 390b6ee45e10487992567c61f5971511 (127.23.153.3:33063)
I20260929 11:26:41.198931 26217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a778cb2abdee40b1b29b0b330698d046" candidate_uuid: "77cf22706b4f487c9d66a93f9bc02804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" is_pre_election: true
I20260929 11:26:41.199294 26217 raft_consensus.cc:2454] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77cf22706b4f487c9d66a93f9bc02804 in term 0.
I20260929 11:26:41.199386 26357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a778cb2abdee40b1b29b0b330698d046" candidate_uuid: "77cf22706b4f487c9d66a93f9bc02804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "390b6ee45e10487992567c61f5971511" is_pre_election: true
I20260929 11:26:41.199647 26357 raft_consensus.cc:2454] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77cf22706b4f487c9d66a93f9bc02804 in term 0.
I20260929 11:26:41.199829 26012 leader_election.cc:302] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e46c2a7ac0341a4acd1de07bb4cbf4c, 77cf22706b4f487c9d66a93f9bc02804; no voters:
I20260929 11:26:41.200148 26504 raft_consensus.cc:2781] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:26:41.200289 26504 raft_consensus.cc:482] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:26:41.200377 26504 raft_consensus.cc:3037] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:41.202600 26504 raft_consensus.cc:504] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:41.203264 26504 leader_election.cc:288] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [CANDIDATE]: Term 1 election: Requested vote from peers 2e46c2a7ac0341a4acd1de07bb4cbf4c (127.23.153.2:45881), 390b6ee45e10487992567c61f5971511 (127.23.153.3:33063)
I20260929 11:26:41.203861 26217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a778cb2abdee40b1b29b0b330698d046" candidate_uuid: "77cf22706b4f487c9d66a93f9bc02804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c"
I20260929 11:26:41.204083 26217 raft_consensus.cc:3037] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:41.204229 26357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a778cb2abdee40b1b29b0b330698d046" candidate_uuid: "77cf22706b4f487c9d66a93f9bc02804" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "390b6ee45e10487992567c61f5971511"
I20260929 11:26:41.204490 26357 raft_consensus.cc:3037] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:41.207289 26217 raft_consensus.cc:2454] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77cf22706b4f487c9d66a93f9bc02804 in term 1.
I20260929 11:26:41.207723 26357 raft_consensus.cc:2454] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77cf22706b4f487c9d66a93f9bc02804 in term 1.
I20260929 11:26:41.207814 26012 leader_election.cc:302] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e46c2a7ac0341a4acd1de07bb4cbf4c, 77cf22706b4f487c9d66a93f9bc02804; no voters:
I20260929 11:26:41.208194 26504 raft_consensus.cc:2781] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:41.208645 26504 raft_consensus.cc:686] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [term 1 LEADER]: Becoming Leader. State: Replica: 77cf22706b4f487c9d66a93f9bc02804, State: Running, Role: LEADER
I20260929 11:26:41.208940 26504 consensus_queue.cc:237] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } }
I20260929 11:26:41.212184 25920 catalog_manager.cc:5495] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77cf22706b4f487c9d66a93f9bc02804 (127.23.153.1). New cstate: current_term: 1 leader_uuid: "77cf22706b4f487c9d66a93f9bc02804" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77cf22706b4f487c9d66a93f9bc02804" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 } health_report { overall_health: UNKNOWN } } }
I20260929 11:26:41.241609 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:41.236048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56622 (local address 127.23.153.1:45795)
0929 11:26:41.236691 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:41.236695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:41.236714 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:26:41.236788 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:41.236791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:41.236861 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:41.236948 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:26:41.237349 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:41.238100 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:41.239485 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:41.239928 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:41.240047 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:41.240684 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:26:41.240855 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:26:41.240967 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:41.241124 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 11:26:41.241218 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":367,"threads_started":1}
I20260929 11:26:41.252817 26357 raft_consensus.cc:1261] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511 [term 1 FOLLOWER]: Refusing update from remote peer 77cf22706b4f487c9d66a93f9bc02804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:41.253204 26217 raft_consensus.cc:1261] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 FOLLOWER]: Refusing update from remote peer 77cf22706b4f487c9d66a93f9bc02804: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:26:41.253562 26504 consensus_queue.cc:1035] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "390b6ee45e10487992567c61f5971511" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:26:41.253855 26542 consensus_queue.cc:1035] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e46c2a7ac0341a4acd1de07bb4cbf4c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 11:26:41.281210 26496 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 11:26:42.885015 25920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:26:42.885257 25920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:26:42.890092 25920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05b1fa39fce34f758dd95f2337655edb: Sending DeleteTablet for 3 replicas of tablet a778cb2abdee40b1b29b0b330698d046
I20260929 11:26:42.890980 26059 tablet_service.cc:1514] Processing DeleteTablet for tablet a778cb2abdee40b1b29b0b330698d046 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:26:42 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:56608
I20260929 11:26:42.890980 26197 tablet_service.cc:1514] Processing DeleteTablet for tablet a778cb2abdee40b1b29b0b330698d046 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:26:42 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:49070
I20260929 11:26:42.891156 26337 tablet_service.cc:1514] Processing DeleteTablet for tablet a778cb2abdee40b1b29b0b330698d046 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:26:42 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:38620
I20260929 11:26:42.895123 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 25993
I20260929 11:26:42.901449 26576 tablet_replica.cc:330] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511: stopping tablet replica
I20260929 11:26:42.901942 26576 raft_consensus.cc:2229] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:26:42.902369 26576 raft_consensus.cc:2258] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:26:42.903911 26576 ts_tablet_manager.cc:1905] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:26:42.911147 26576 ts_tablet_manager.cc:1918] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 11:26:42.911684 26575 tablet_replica.cc:330] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804: stopping tablet replica
I20260929 11:26:42.911938 26576 log.cc:1196] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/a778cb2abdee40b1b29b0b330698d046
I20260929 11:26:42.911952 26575 raft_consensus.cc:2229] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:26:42.912016 26574 tablet_replica.cc:330] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: stopping tablet replica
I20260929 11:26:42.912226 26574 raft_consensus.cc:2229] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:26:42.912355 26575 raft_consensus.cc:2258] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:26:42.912531 26574 raft_consensus.cc:2258] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:26:42.912671 26576 ts_tablet_manager.cc:1939] T a778cb2abdee40b1b29b0b330698d046 P 390b6ee45e10487992567c61f5971511: Deleting consensus metadata
I20260929 11:26:42.913538 26575 ts_tablet_manager.cc:1905] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:26:42.913712 26574 ts_tablet_manager.cc:1905] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 11:26:42 dist-test-slave-qk05 krb5kdc[25877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681200, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:26:42.914165 25909 catalog_manager.cc:4841] TS 390b6ee45e10487992567c61f5971511 (127.23.153.3:33063): tablet a778cb2abdee40b1b29b0b330698d046 (table test-table [id=f7dcd0b55f5c4b5ba03221d572044654]) successfully deleted
I20260929 11:26:42.919780 26575 ts_tablet_manager.cc:1918] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 11:26:42.919925 26575 log.cc:1196] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/a778cb2abdee40b1b29b0b330698d046
I20260929 11:26:42.920318 26575 ts_tablet_manager.cc:1939] T a778cb2abdee40b1b29b0b330698d046 P 77cf22706b4f487c9d66a93f9bc02804: Deleting consensus metadata
I20260929 11:26:42.921471 25907 catalog_manager.cc:4841] TS 77cf22706b4f487c9d66a93f9bc02804 (127.23.153.1:45795): tablet a778cb2abdee40b1b29b0b330698d046 (table test-table [id=f7dcd0b55f5c4b5ba03221d572044654]) successfully deleted
I20260929 11:26:42.921649 26574 ts_tablet_manager.cc:1918] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 11:26:42.921797 26574 log.cc:1196] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/a778cb2abdee40b1b29b0b330698d046
I20260929 11:26:42.922127 26574 ts_tablet_manager.cc:1939] T a778cb2abdee40b1b29b0b330698d046 P 2e46c2a7ac0341a4acd1de07bb4cbf4c: Deleting consensus metadata
I20260929 11:26:42.923339 25909 catalog_manager.cc:4841] TS 2e46c2a7ac0341a4acd1de07bb4cbf4c (127.23.153.2:45881): tablet a778cb2abdee40b1b29b0b330698d046 (table test-table [id=f7dcd0b55f5c4b5ba03221d572044654]) successfully deleted
I20260929 11:26:42.924009 26578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:42.896248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.23.153.1:45795)
0929 11:26:42.903224 (+ 6976us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:42.903229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:42.903260 (+ 31us) server_negotiation.cc:407] Connection header received
0929 11:26:42.903333 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:42.903339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:42.903432 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:42.903640 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:42.905697 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:42.906855 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:42.908793 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:42.909490 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:42.909601 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:42.915525 (+ 5924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:42.915557 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:42.915586 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:42.915663 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:42.920738 (+ 5075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:42.921763 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:42.921773 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:42.921777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:42.921951 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:42.922357 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:42.922362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:42.922366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:42.922742 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:42.923059 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:42.923419 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 11:26:42.923665 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6808,"thread_start_us":6609,"threads_started":1}
W20260929 11:26:42.924688 26123 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56632
W20260929 11:26:42.925338 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 25993: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:26:42.925549 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 25993
I20260929 11:26:42.945497 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 26133
Sep 29 11:26:42 dist-test-slave-qk05 krb5kdc[25877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681200, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.153.2@KRBTEST.COM
I20260929 11:26:42.961961 26579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:42.946335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49112 (local address 127.23.153.2:45881)
0929 11:26:42.946812 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:42.946817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:42.946834 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:26:42.946889 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:42.946892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:42.946959 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:42.947086 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:42.948313 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:42.949073 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:42.950412 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:42.950949 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:42.951122 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:42.955632 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:42.955654 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:42.955672 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:42.955719 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:42.958910 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:42.959741 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:42.959749 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:42.959751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:42.959904 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:42.960272 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:42.960275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:42.960277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:42.960662 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:42.960920 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:42.961484 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 11:26:42.961674 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":235,"threads_started":1}
W20260929 11:26:42.962688 26263 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49112
W20260929 11:26:42.963390 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 26133: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:26:42.963536 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 26133
I20260929 11:26:42.987195 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 26272
Sep 29 11:26:42 dist-test-slave-qk05 krb5kdc[25877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681200, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.153.3@KRBTEST.COM
I20260929 11:26:43.007546 26580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:42.988174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38634 (local address 127.23.153.3:33063)
0929 11:26:42.988810 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:42.988815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:42.988833 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:26:42.988888 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:42.988892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:42.988955 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:42.989059 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:42.990459 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:42.991272 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:42.992592 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:42.992997 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:42.993113 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:42.999057 (+ 5944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:42.999090 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:42.999109 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:42.999157 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:43.003728 (+ 4571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:43.004785 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:43.004795 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:43.004798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:43.004977 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:43.005547 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:43.005554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:43.005558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:43.006098 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:43.006495 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:43.006896 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 11:26:43.007206 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":322,"threads_started":1}
W20260929 11:26:43.008424 26402 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38634
W20260929 11:26:43.009055 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 26272: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:26:43.009462 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 26272
I20260929 11:26:43.029146 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 25890
I20260929 11:26:43.030525 25962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:43.154322 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 25890
2026-09-29T11:26:43Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8551 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:26:43 dist-test-slave-qk05 krb5kdc[26584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:26:43 dist-test-slave-qk05 krb5kdc[26584](info): set up 2 sockets
Sep 29 11:26:43 dist-test-slave-qk05 krb5kdc[26584](info): commencing operation
krb5kdc: starting...
W20260929 11:26:44.187137 24164 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:26:44 dist-test-slave-qk05 krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:26:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:26:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:26:44.378880 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:42085
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:42085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:44.730664 26597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:44.730981 26597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:44.731132 26597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:44.740331 26597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:44.740481 26597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:44.740530 26597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:44.740609 26597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:44.740787 26597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:44.751466 26597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:42085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:42085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:44.752840 26597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:44.754231 26597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:44.767970 26602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:44.768040 26603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:44.768004 26605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:44.769354 26597 server_base.cc:1034] running on GCE node
I20260929 11:26:44.770327 26597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:44.772094 26597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:44.773370 26597 hybrid_clock.cc:648] HybridClock initialized: now 1790681204773316 us; error 37 us; skew 500 ppm
I20260929 11:26:44.773785 26597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:44 dist-test-slave-qk05 krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681204, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:44.777395 26597 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:26:44.779078 26597 webserver.cc:460] Webserver started at http://127.23.153.62:44179/ using document root <none> and password file <none>
I20260929 11:26:44.779803 26597 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:44.779940 26597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:44.780246 26597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:44.783370 26597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "452a367a66b24cd580bbb14bb889c468"
format_stamp: "Formatted at 2026-09-29 11:26:44 on dist-test-slave-qk05"
server_key: "3a4e097547db05991f57cb9ebab0540b"
server_key_iv: "0be5ba9dacd0ef6207a89bf49414b25c"
server_key_version: "encryptionkey@0"
I20260929 11:26:44.784134 26597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "452a367a66b24cd580bbb14bb889c468"
format_stamp: "Formatted at 2026-09-29 11:26:44 on dist-test-slave-qk05"
server_key: "3a4e097547db05991f57cb9ebab0540b"
server_key_iv: "0be5ba9dacd0ef6207a89bf49414b25c"
server_key_version: "encryptionkey@0"
I20260929 11:26:44.789645 26597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:26:44.793493 26612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:44.795147 26597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:26:44.795341 26597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "452a367a66b24cd580bbb14bb889c468"
format_stamp: "Formatted at 2026-09-29 11:26:44 on dist-test-slave-qk05"
server_key: "3a4e097547db05991f57cb9ebab0540b"
server_key_iv: "0be5ba9dacd0ef6207a89bf49414b25c"
server_key_version: "encryptionkey@0"
I20260929 11:26:44.795554 26597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:44.812167 26597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:44.816565 26597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:44.816844 26597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:44.836189 26597 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:42085
I20260929 11:26:44.836208 26663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:42085 every 8 connection(s)
I20260929 11:26:44.837739 26597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:26:44.841768 26664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:44.848225 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 26597
I20260929 11:26:44.848433 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:26:44.849145 24164 external_mini_cluster.cc:1428] Setting key 1064235f6df12fb3357de1b4909a7e21
I20260929 11:26:44.853938 26664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468: Bootstrap starting.
I20260929 11:26:44.858143 26664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:44.859673 26664 log.cc:824] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:44.863545 26664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468: No bootstrap required, opened a new log
Sep 29 11:26:44 dist-test-slave-qk05 krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:44.874387 26664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452a367a66b24cd580bbb14bb889c468" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42085 } }
I20260929 11:26:44.874963 26664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:44.875142 26664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 452a367a66b24cd580bbb14bb889c468, State: Initialized, Role: FOLLOWER
I20260929 11:26:44.875881 26664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452a367a66b24cd580bbb14bb889c468" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42085 } }
I20260929 11:26:44.876166 26664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:26:44.876363 26664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:26:44.876556 26664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:44.878444 26667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:44.851528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34850 (local address 127.23.153.62:42085)
0929 11:26:44.852382 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:44.852400 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:44.852459 (+ 59us) server_negotiation.cc:407] Connection header received
0929 11:26:44.855161 (+ 2702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:44.855197 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:44.855779 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:44.856350 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:44.858025 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:44.860088 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:44.861846 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:44.862940 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:44.863207 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:44.867717 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:44.867731 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:44.867758 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:44.867815 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:44.872299 (+ 4484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:44.873524 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:44.873536 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:44.873546 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:44.873926 (+ 380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:44.874412 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:44.874417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:44.874421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:44.875252 (+ 831us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:44.875567 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:44.876369 (+ 802us) server_negotiation.cc:299] Negotiation successful
0929 11:26:44.876893 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":343,"threads_started":1}
I20260929 11:26:44.879769 26664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452a367a66b24cd580bbb14bb889c468" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42085 } }
I20260929 11:26:44.880581 26664 leader_election.cc:302] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 452a367a66b24cd580bbb14bb889c468; no voters:
I20260929 11:26:44.881358 26664 leader_election.cc:288] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:26:44.881564 26669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:44.882524 26669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468 [term 1 LEADER]: Becoming Leader. State: Replica: 452a367a66b24cd580bbb14bb889c468, State: Running, Role: LEADER
I20260929 11:26:44.883237 26669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452a367a66b24cd580bbb14bb889c468" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42085 } }
I20260929 11:26:44.883755 26664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:26:44.886144 26671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 452a367a66b24cd580bbb14bb889c468. Latest consensus state: current_term: 1 leader_uuid: "452a367a66b24cd580bbb14bb889c468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452a367a66b24cd580bbb14bb889c468" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42085 } } }
I20260929 11:26:44.886590 26671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:44.886940 26670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "452a367a66b24cd580bbb14bb889c468" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452a367a66b24cd580bbb14bb889c468" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42085 } } }
I20260929 11:26:44.887274 26670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:44.889204 26675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:26:44.895056 26675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:26:44.907263 26675 catalog_manager.cc:1347] Generated new cluster ID: 0a0b341fbb86491d8425d65f3ea88c02
I20260929 11:26:44.907438 26675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:26:44.932068 26675 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:26:44.933609 26675 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:26:44.951987 26675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 452a367a66b24cd580bbb14bb889c468: Generated new TSK 0
I20260929 11:26:44.953284 26675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
I20260929 11:26:45.018810 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42085
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:45.389616 26690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:45.389933 26690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:45.390058 26690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:45.399549 26690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:45.399729 26690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:45.399902 26690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:26:45.413218 26690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:45.414919 26690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:45.416882 26690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:45.431778 26695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:45.431670 26696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:45.431706 26698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:45.432701 26690 server_base.cc:1034] running on GCE node
I20260929 11:26:45.433411 26690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:45.434471 26690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:45.435701 26690 hybrid_clock.cc:648] HybridClock initialized: now 1790681205435644 us; error 61 us; skew 500 ppm
I20260929 11:26:45.436064 26690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:45 dist-test-slave-qk05 krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681205, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:45.439409 26690 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:26:45.441023 26690 webserver.cc:460] Webserver started at http://127.23.153.1:32825/ using document root <none> and password file <none>
I20260929 11:26:45.441774 26690 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:45.441907 26690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:45.442215 26690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:45.445701 26690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "abe97232ddef4007b28c5a6c6152ba5d"
format_stamp: "Formatted at 2026-09-29 11:26:45 on dist-test-slave-qk05"
server_key: "0e033cea41186aa13f37d0aec1db64f4"
server_key_iv: "1be1bbde2509cd2dfdbd35675ed17ac4"
server_key_version: "encryptionkey@0"
I20260929 11:26:45.446676 26690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "abe97232ddef4007b28c5a6c6152ba5d"
format_stamp: "Formatted at 2026-09-29 11:26:45 on dist-test-slave-qk05"
server_key: "0e033cea41186aa13f37d0aec1db64f4"
server_key_iv: "1be1bbde2509cd2dfdbd35675ed17ac4"
server_key_version: "encryptionkey@0"
I20260929 11:26:45.452767 26690 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260929 11:26:45.456884 26705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:45.458501 26690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:26:45.458704 26690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "abe97232ddef4007b28c5a6c6152ba5d"
format_stamp: "Formatted at 2026-09-29 11:26:45 on dist-test-slave-qk05"
server_key: "0e033cea41186aa13f37d0aec1db64f4"
server_key_iv: "1be1bbde2509cd2dfdbd35675ed17ac4"
server_key_version: "encryptionkey@0"
I20260929 11:26:45.458928 26690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:45.479488 26690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:45.484093 26690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:45.484422 26690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:45.485548 26690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:26:45.487031 26690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:45.487118 26690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:45.487246 26690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:45.487298 26690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:45.520630 26690 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:45801
I20260929 11:26:45.520697 26817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:45801 every 8 connection(s)
I20260929 11:26:45.522056 26690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:26:45.532106 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 26690
I20260929 11:26:45.532337 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:26:45.532940 24164 external_mini_cluster.cc:1428] Setting key 242916c06b32408b151dfa84ebf14ede
Sep 29 11:26:45 dist-test-slave-qk05 krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681205, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:45.544394 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:45.525031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:52803 (local address 127.23.153.62:42085)
0929 11:26:45.525504 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:45.525509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:45.526617 (+ 1108us) server_negotiation.cc:407] Connection header received
0929 11:26:45.528781 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:45.528787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:45.528894 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:45.528995 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:45.530793 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:45.531585 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:45.533458 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:45.534036 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:45.534173 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:45.538435 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:45.538441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:45.538444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:45.538484 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:45.541134 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:45.541984 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:45.541990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:45.541992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:45.542119 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:45.542555 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:45.542560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:45.542562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:45.542841 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:45.542985 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:45.543785 (+ 800us) server_negotiation.cc:299] Negotiation successful
0929 11:26:45.544044 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":241,"threads_started":1}
I20260929 11:26:45.545465 26821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:45.525532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42085 (local address 127.23.153.1:52803)
0929 11:26:45.526377 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:45.526431 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:45.528403 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:45.529296 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:45.529307 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:45.529931 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:45.530572 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:45.530593 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:45.531766 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:45.531770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:45.533224 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:45.533241 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:45.534307 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:45.534315 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:45.534581 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:45.535361 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:45.535391 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:45.538122 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:45.541331 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:45.541340 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:45.541357 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:45.541843 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:45.542232 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:45.542237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:45.542240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:45.542441 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:45.542985 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:45.542995 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:45.543397 (+ 402us) client_negotiation.cc:769] Sending connection context
0929 11:26:45.543666 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 11:26:45.544266 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":296,"threads_started":1}
I20260929 11:26:45.547749 26818 heartbeater.cc:347] Connected to a master server at 127.23.153.62:42085
I20260929 11:26:45.548106 26818 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:45.548975 26818 heartbeater.cc:511] Master 127.23.153.62:42085 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.153.2@KRBTEST.COM; defaulting to no policy
I20260929 11:26:45.552232 26629 ts_manager.cc:194] Registered new tserver with Master: abe97232ddef4007b28c5a6c6152ba5d (127.23.153.1:45801)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.2@KRBTEST.COM" created.
I20260929 11:26:45.554471 26629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:52803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
I20260929 11:26:45.578260 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42085
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:45.952543 26824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:45.952867 26824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:45.952975 26824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:45.961398 26824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:45.961546 26824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:45.961663 26824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:26:45.972362 26824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:45.974643 26824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:45.976300 26824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:45.990908 26832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:45.990885 26830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:45.990944 26829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:45.992254 26824 server_base.cc:1034] running on GCE node
I20260929 11:26:45.992942 26824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:45.994132 26824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:45.995390 26824 hybrid_clock.cc:648] HybridClock initialized: now 1790681205995324 us; error 65 us; skew 500 ppm
I20260929 11:26:45.995870 26824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:45 dist-test-slave-qk05 krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681205, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:46.003165 26824 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:26:46.005321 26824 webserver.cc:460] Webserver started at http://127.23.153.2:46077/ using document root <none> and password file <none>
I20260929 11:26:46.006140 26824 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:46.006278 26824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:46.006618 26824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:46.010118 26824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "121f4ca1b2aa44f994d4ad65d55d1b62"
format_stamp: "Formatted at 2026-09-29 11:26:46 on dist-test-slave-qk05"
server_key: "e9245e4c49fdef053647d1547c128374"
server_key_iv: "b625dfce624ce4db9e2daffdea825e8c"
server_key_version: "encryptionkey@0"
I20260929 11:26:46.011240 26824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "121f4ca1b2aa44f994d4ad65d55d1b62"
format_stamp: "Formatted at 2026-09-29 11:26:46 on dist-test-slave-qk05"
server_key: "e9245e4c49fdef053647d1547c128374"
server_key_iv: "b625dfce624ce4db9e2daffdea825e8c"
server_key_version: "encryptionkey@0"
I20260929 11:26:46.017635 26824 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:26:46.021744 26839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:46.023461 26824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 11:26:46.023761 26824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "121f4ca1b2aa44f994d4ad65d55d1b62"
format_stamp: "Formatted at 2026-09-29 11:26:46 on dist-test-slave-qk05"
server_key: "e9245e4c49fdef053647d1547c128374"
server_key_iv: "b625dfce624ce4db9e2daffdea825e8c"
server_key_version: "encryptionkey@0"
I20260929 11:26:46.024010 26824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:46.049890 26824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:46.054466 26824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:46.054776 26824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:46.055882 26824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:26:46.057600 26824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:46.057691 26824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:46.057813 26824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:46.057868 26824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:46.093377 26824 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:44445
I20260929 11:26:46.093428 26951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:44445 every 8 connection(s)
I20260929 11:26:46.094792 26824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:26:46.101706 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 26824
I20260929 11:26:46.101894 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:26:46.102303 24164 external_mini_cluster.cc:1428] Setting key c30e746663d7c52f1c6dfb7e5638a95e
Sep 29 11:26:46 dist-test-slave-qk05 krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681205, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:46.119218 26954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:46.097668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:60675 (local address 127.23.153.62:42085)
0929 11:26:46.098091 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:46.098096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:46.099091 (+ 995us) server_negotiation.cc:407] Connection header received
0929 11:26:46.101143 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:46.101149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:46.101246 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:46.101340 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:46.103215 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:46.104029 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:46.106061 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:46.106620 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:46.106940 (+ 320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:46.112494 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:46.112500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:46.112504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:46.112546 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:46.115385 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:46.116322 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:46.116328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:46.116330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:46.116477 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:46.116955 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:46.116959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:46.116961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:46.117275 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:46.117448 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:46.118655 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0929 11:26:46.118877 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":221,"threads_started":1}
I20260929 11:26:46.120772 26955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:46.098072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42085 (local address 127.23.153.2:60675)
0929 11:26:46.098899 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:46.098941 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:46.100834 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:46.101624 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:46.101635 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:46.102261 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:46.102987 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:46.103051 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:46.104186 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:46.104191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:46.105740 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:46.105772 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:46.107095 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:46.107105 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:46.107385 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:46.108569 (+ 1184us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:46.108595 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:46.112160 (+ 3565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:46.115596 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:46.115607 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:46.115625 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:46.116158 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:46.116599 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:46.116604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:46.116609 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:46.116829 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:46.117491 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:46.117501 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:46.118064 (+ 563us) client_negotiation.cc:769] Sending connection context
0929 11:26:46.118447 (+ 383us) client_negotiation.cc:240] Negotiation successful
0929 11:26:46.119157 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":292,"threads_started":1}
WARNING: no policy specified for kudu/127.23.153.3@KRBTEST.COM; defaulting to no policy
I20260929 11:26:46.122627 26952 heartbeater.cc:347] Connected to a master server at 127.23.153.62:42085
I20260929 11:26:46.122995 26952 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:46.124449 26952 heartbeater.cc:511] Master 127.23.153.62:42085 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.3@KRBTEST.COM" created.
I20260929 11:26:46.126363 26629 ts_manager.cc:194] Registered new tserver with Master: 121f4ca1b2aa44f994d4ad65d55d1b62 (127.23.153.2:44445)
I20260929 11:26:46.127285 26629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:60675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
I20260929 11:26:46.146767 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42085
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:46.494613 26958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:46.494966 26958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:46.495111 26958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:46.502795 26958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:46.502944 26958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:46.503121 26958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:26:46.516067 26958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:46.517499 26958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:46.518880 26958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:46.532796 26963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:46.532775 26964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:46.532842 26966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:46.533596 26958 server_base.cc:1034] running on GCE node
I20260929 11:26:46.534193 26958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:46.535207 26958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:46.536438 26958 hybrid_clock.cc:648] HybridClock initialized: now 1790681206536380 us; error 49 us; skew 500 ppm
I20260929 11:26:46.536783 26958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:46 dist-test-slave-qk05 krb5kdc[26584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681206, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:46.540139 26958 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:26:46.541868 26958 webserver.cc:460] Webserver started at http://127.23.153.3:46437/ using document root <none> and password file <none>
I20260929 11:26:46.542613 26958 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:46.542742 26958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:46.543092 26958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:46.546223 26958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "16634dc32fa049929e42145880863739"
format_stamp: "Formatted at 2026-09-29 11:26:46 on dist-test-slave-qk05"
server_key: "86255c4a572392568a5761a64236e00d"
server_key_iv: "210ff79f657b69ee9932d23ed0be3e83"
server_key_version: "encryptionkey@0"
I20260929 11:26:46.547190 26958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "16634dc32fa049929e42145880863739"
format_stamp: "Formatted at 2026-09-29 11:26:46 on dist-test-slave-qk05"
server_key: "86255c4a572392568a5761a64236e00d"
server_key_iv: "210ff79f657b69ee9932d23ed0be3e83"
server_key_version: "encryptionkey@0"
I20260929 11:26:46.552966 26958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:26:46.556984 26973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:46.558547 26958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:26:46.558804 26818 heartbeater.cc:503] Master 127.23.153.62:42085 was elected leader, sending a full tablet report...
I20260929 11:26:46.558758 26958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "16634dc32fa049929e42145880863739"
format_stamp: "Formatted at 2026-09-29 11:26:46 on dist-test-slave-qk05"
server_key: "86255c4a572392568a5761a64236e00d"
server_key_iv: "210ff79f657b69ee9932d23ed0be3e83"
server_key_version: "encryptionkey@0"
I20260929 11:26:46.559089 26958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:46.573463 26958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:46.578052 26958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:46.578343 26958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:46.579357 26958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:26:46.580824 26958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:46.580907 26958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:46.581025 26958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:46.581076 26958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:46.613076 26958 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:43345
I20260929 11:26:46.613121 27085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:43345 every 8 connection(s)
I20260929 11:26:46.614388 26958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:26:46.616575 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 26958
I20260929 11:26:46.616776 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:26:46.617322 24164 external_mini_cluster.cc:1428] Setting key ac0f76607d09b87ca07d4b8c681cca27
Sep 29 11:26:46 dist-test-slave-qk05 krb5kdc[26584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681206, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:46.635659 26954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:46.617117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:36563 (local address 127.23.153.62:42085)
0929 11:26:46.617287 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:46.617292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:46.618893 (+ 1601us) server_negotiation.cc:407] Connection header received
0929 11:26:46.620798 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:46.620802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:46.620872 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:46.620953 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:46.622525 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:46.623356 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:46.624969 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:46.625371 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:46.625466 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:46.629416 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:46.629422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:46.629426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:46.629467 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:46.632408 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:46.633224 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:46.633228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:46.633231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:46.633365 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:46.633817 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:46.633820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:46.633823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:46.634119 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:46.634253 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:46.635066 (+ 813us) server_negotiation.cc:299] Negotiation successful
0929 11:26:46.635309 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:26:46.636686 27088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:46.617836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42085 (local address 127.23.153.3:36563)
0929 11:26:46.618720 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:46.618776 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:46.620536 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:46.621165 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:46.621175 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:46.621739 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:46.622355 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:46.622375 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:46.623520 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:46.623524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:46.624838 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:46.624852 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:46.625588 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:46.625593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:46.625815 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:46.626476 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:46.626502 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:46.628851 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:46.632589 (+ 3738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:46.632600 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:46.632618 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:46.633096 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:46.633471 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:46.633476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:46.633480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:46.633698 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:46.634247 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:46.634256 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:46.634663 (+ 407us) client_negotiation.cc:769] Sending connection context
0929 11:26:46.634918 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 11:26:46.635524 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":378,"threads_started":1}
I20260929 11:26:46.638278 27086 heartbeater.cc:347] Connected to a master server at 127.23.153.62:42085
I20260929 11:26:46.638624 27086 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:46.639428 27086 heartbeater.cc:511] Master 127.23.153.62:42085 requested a full tablet report, sending...
I20260929 11:26:46.640937 26629 ts_manager.cc:194] Registered new tserver with Master: 16634dc32fa049929e42145880863739 (127.23.153.3:43345)
I20260929 11:26:46.641741 26629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:36563
I20260929 11:26:46.645368 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 11:26:46.676136 26954 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:26:46.673580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34860 (local address 127.23.153.62:42085)
0929 11:26:46.673800 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:46.673807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:46.674144 (+ 337us) server_negotiation.cc:407] Connection header received
0929 11:26:46.674292 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:46.674296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:46.674364 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:46.674452 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:46.675810 (+ 1358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34860: BlockingRecv error: recv got EOF from 127.0.0.1:34860 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 11:26:46.679066 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 26690
W20260929 11:26:46.681898 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 26690: Not authorized: Client connection negotiation failed: client connection to 127.23.153.1:45801: server requires authentication, but client does not have Kerberos credentials available
I20260929 11:26:46.682118 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 26690
I20260929 11:26:46.697240 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 26824
W20260929 11:26:46.699957 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 26824: Not authorized: Client connection negotiation failed: client connection to 127.23.153.2:44445: server requires authentication, but client does not have Kerberos credentials available
I20260929 11:26:46.700131 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 26824
I20260929 11:26:46.712673 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 26958
W20260929 11:26:46.715906 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 26958: Not authorized: Client connection negotiation failed: client connection to 127.23.153.3:43345: server requires authentication, but client does not have Kerberos credentials available
I20260929 11:26:46.716053 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 26958
I20260929 11:26:46.729038 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 26597
I20260929 11:26:46.730473 26659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:46.841463 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 26597
2026-09-29T11:26:46Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3685 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:26:46 dist-test-slave-qk05 krb5kdc[27103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:26:46 dist-test-slave-qk05 krb5kdc[27103](info): set up 2 sockets
Sep 29 11:26:46 dist-test-slave-qk05 krb5kdc[27103](info): commencing operation
krb5kdc: starting...
W20260929 11:26:47.859649 24164 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:26:47 dist-test-slave-qk05 krb5kdc[27103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:26:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:26:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:26:48.058162 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:45733
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:41339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:45733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:48.416740 27116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:48.417068 27116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:48.417186 27116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:48.425113 27116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:48.425254 27116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:48.425302 27116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:48.425364 27116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:48.425428 27116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:48.437579 27116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:41339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:45733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:45733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:48.439208 27116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:48.440765 27116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:48.453899 27122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:48.453915 27121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:48.454025 27124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:48.454859 27116 server_base.cc:1034] running on GCE node
I20260929 11:26:48.455765 27116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:48.457398 27116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:48.458683 27116 hybrid_clock.cc:648] HybridClock initialized: now 1790681208458623 us; error 61 us; skew 500 ppm
I20260929 11:26:48.459159 27116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:48 dist-test-slave-qk05 krb5kdc[27103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681208, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:48.462751 27116 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:26:48.465153 27116 webserver.cc:460] Webserver started at http://127.23.153.62:43939/ using document root <none> and password file <none>
I20260929 11:26:48.466286 27116 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:48.466491 27116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:48.466857 27116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:48.471001 27116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "e0110e99a7c24bcfb5109f9c35c0df7c"
format_stamp: "Formatted at 2026-09-29 11:26:48 on dist-test-slave-qk05"
server_key: "55a5534b06a2b376b22cf1df2f6ae36c"
server_key_iv: "d138b0d9399f64a6f04ca4a87fb2b058"
server_key_version: "encryptionkey@0"
I20260929 11:26:48.472024 27116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "e0110e99a7c24bcfb5109f9c35c0df7c"
format_stamp: "Formatted at 2026-09-29 11:26:48 on dist-test-slave-qk05"
server_key: "55a5534b06a2b376b22cf1df2f6ae36c"
server_key_iv: "d138b0d9399f64a6f04ca4a87fb2b058"
server_key_version: "encryptionkey@0"
I20260929 11:26:48.479187 27116 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260929 11:26:48.483327 27131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:48.485015 27116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:26:48.485231 27116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "e0110e99a7c24bcfb5109f9c35c0df7c"
format_stamp: "Formatted at 2026-09-29 11:26:48 on dist-test-slave-qk05"
server_key: "55a5534b06a2b376b22cf1df2f6ae36c"
server_key_iv: "d138b0d9399f64a6f04ca4a87fb2b058"
server_key_version: "encryptionkey@0"
I20260929 11:26:48.485457 27116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:48.513908 27116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:48.518635 27116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:48.518925 27116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:48.539072 27116 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:45733
I20260929 11:26:48.539149 27182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:45733 every 8 connection(s)
I20260929 11:26:48.540536 27116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:26:48.544756 27183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:48.549538 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 27116
I20260929 11:26:48.549768 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:26:48.550391 24164 external_mini_cluster.cc:1428] Setting key 7f8f79612c88995c9806dbf50540c946
I20260929 11:26:48.558089 27183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c: Bootstrap starting.
I20260929 11:26:48.562692 27183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:48.564021 27183 log.cc:824] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c: Log is configured to *not* fsync() on all Append() calls
Sep 29 11:26:48 dist-test-slave-qk05 krb5kdc[27103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:48.568158 27183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c: No bootstrap required, opened a new log
I20260929 11:26:48.577765 27183 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0110e99a7c24bcfb5109f9c35c0df7c" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45733 } }
I20260929 11:26:48.578158 27183 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:48.578262 27183 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0110e99a7c24bcfb5109f9c35c0df7c, State: Initialized, Role: FOLLOWER
I20260929 11:26:48.578294 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:48.552763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43670 (local address 127.23.153.62:45733)
0929 11:26:48.553718 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:48.553730 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:48.553775 (+ 45us) server_negotiation.cc:407] Connection header received
0929 11:26:48.555800 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:48.555829 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:48.556299 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:48.556710 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:48.558711 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:48.560734 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:48.562616 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:48.563505 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:48.563665 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:48.568756 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:48.568766 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:48.568783 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:48.568829 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:48.572406 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:48.573593 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:48.573610 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:48.573622 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:48.573993 (+ 371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:48.574827 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:48.574835 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:48.574839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:48.575806 (+ 967us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:48.576080 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:48.576672 (+ 592us) server_negotiation.cc:299] Negotiation successful
0929 11:26:48.577103 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":421,"threads_started":1}
I20260929 11:26:48.578977 27183 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0110e99a7c24bcfb5109f9c35c0df7c" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45733 } }
I20260929 11:26:48.579391 27183 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:26:48.579669 27183 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:26:48.579918 27183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:48.583375 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:26:48.583703 27183 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0110e99a7c24bcfb5109f9c35c0df7c" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45733 } }
I20260929 11:26:48.584265 27183 leader_election.cc:302] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0110e99a7c24bcfb5109f9c35c0df7c; no voters:
I20260929 11:26:48.584856 27183 leader_election.cc:288] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:26:48.585018 27188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:48.586956 27183 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:26:48.592231 27188 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c [term 1 LEADER]: Becoming Leader. State: Replica: e0110e99a7c24bcfb5109f9c35c0df7c, State: Running, Role: LEADER
I20260929 11:26:48.593151 27188 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0110e99a7c24bcfb5109f9c35c0df7c" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45733 } }
I20260929 11:26:48.601544 27189 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0110e99a7c24bcfb5109f9c35c0df7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0110e99a7c24bcfb5109f9c35c0df7c" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45733 } } }
I20260929 11:26:48.602224 27189 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:48.601563 27190 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0110e99a7c24bcfb5109f9c35c0df7c. Latest consensus state: current_term: 1 leader_uuid: "e0110e99a7c24bcfb5109f9c35c0df7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0110e99a7c24bcfb5109f9c35c0df7c" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45733 } } }
I20260929 11:26:48.602514 27190 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c [sys.catalog]: This master's current role is: LEADER
W20260929 11:26:48.607647 27205 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:26:48.607806 27205 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:26:48.607937 27206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:26:48.609829 27206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:26:48.622130 27206 catalog_manager.cc:1347] Generated new cluster ID: 5cec2525c7d44583ae5addd553712785
I20260929 11:26:48.622323 27206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:26:48.633846 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:48.623260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43672 (local address 127.23.153.62:45733)
0929 11:26:48.623469 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:48.623476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:48.624070 (+ 594us) server_negotiation.cc:407] Connection header received
0929 11:26:48.624267 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:48.624274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:48.624345 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:48.624431 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:48.625873 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:48.626937 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:48.629211 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:48.629722 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:48.629865 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:48.631322 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:48.631333 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:48.631338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:26:48.632192 (+ 854us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:48.632473 (+ 281us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:26:48.632551 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:48.632746 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:48.633040 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 11:26:48.633190 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 11:26:48.637308 27206 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:26:48.639122 27206 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:26:48.647250 27206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0110e99a7c24bcfb5109f9c35c0df7c: Generated new TSK 0
I20260929 11:26:48.648546 27206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:26:48.664058 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 27116
I20260929 11:26:48.665570 27178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:48.759996 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 27116
2026-09-29T11:26:48Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1913 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:26:48 dist-test-slave-qk05 krb5kdc[27218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:26:48 dist-test-slave-qk05 krb5kdc[27218](info): set up 2 sockets
Sep 29 11:26:48 dist-test-slave-qk05 krb5kdc[27218](info): commencing operation
krb5kdc: starting...
W20260929 11:26:49.779347 24164 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:26:49 dist-test-slave-qk05 krb5kdc[27218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:26:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:26:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:26:49.978468 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:41449
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:41449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:50.323799 27231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:50.324153 27231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:50.324283 27231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:50.333388 27231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:50.333565 27231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:50.333639 27231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:50.333721 27231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:50.333784 27231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:50.344709 27231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:41449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:41449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:50.346788 27231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:50.348618 27231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:50.362375 27236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:50.362418 27239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:50.362449 27237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:50.363344 27231 server_base.cc:1034] running on GCE node
I20260929 11:26:50.364241 27231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:50.365904 27231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:50.367161 27231 hybrid_clock.cc:648] HybridClock initialized: now 1790681210367084 us; error 69 us; skew 500 ppm
I20260929 11:26:50.367545 27231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:50 dist-test-slave-qk05 krb5kdc[27218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681210, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:50.372013 27231 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:26:50.374125 27231 webserver.cc:460] Webserver started at http://127.23.153.62:36795/ using document root <none> and password file <none>
I20260929 11:26:50.375083 27231 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:50.375375 27231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:50.375954 27231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:50.381075 27231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "e40ee61a2c5148db8444ceaba60b0544"
format_stamp: "Formatted at 2026-09-29 11:26:50 on dist-test-slave-qk05"
server_key: "161bed9bff020d0546477e132316e3d0"
server_key_iv: "5283fdb55d4216cbe0c504f2e96c400c"
server_key_version: "encryptionkey@0"
I20260929 11:26:50.382177 27231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "e40ee61a2c5148db8444ceaba60b0544"
format_stamp: "Formatted at 2026-09-29 11:26:50 on dist-test-slave-qk05"
server_key: "161bed9bff020d0546477e132316e3d0"
server_key_iv: "5283fdb55d4216cbe0c504f2e96c400c"
server_key_version: "encryptionkey@0"
I20260929 11:26:50.389479 27231 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260929 11:26:50.393823 27246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:50.395694 27231 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:26:50.395901 27231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "e40ee61a2c5148db8444ceaba60b0544"
format_stamp: "Formatted at 2026-09-29 11:26:50 on dist-test-slave-qk05"
server_key: "161bed9bff020d0546477e132316e3d0"
server_key_iv: "5283fdb55d4216cbe0c504f2e96c400c"
server_key_version: "encryptionkey@0"
I20260929 11:26:50.396131 27231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:50.415652 27231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:50.421844 27231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:50.422197 27231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:50.442616 27231 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:41449
I20260929 11:26:50.442631 27297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:41449 every 8 connection(s)
I20260929 11:26:50.444175 27231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:26:50.448401 27298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:50.448969 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 27231
I20260929 11:26:50.449149 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:26:50.449788 24164 external_mini_cluster.cc:1428] Setting key 3c31c7b1d528272f6c6d5439093cc9fa
I20260929 11:26:50.464025 27298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544: Bootstrap starting.
Sep 29 11:26:50 dist-test-slave-qk05 krb5kdc[27218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:50.467651 27298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:50.469015 27298 log.cc:824] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:50.472992 27298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544: No bootstrap required, opened a new log
I20260929 11:26:50.478266 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:50.452166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.23.153.62:41449)
0929 11:26:50.453056 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:50.453076 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:50.453141 (+ 65us) server_negotiation.cc:407] Connection header received
0929 11:26:50.455515 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:50.455557 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:50.456268 (+ 711us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:50.456850 (+ 582us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:50.458818 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:50.460970 (+ 2152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:50.462730 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:50.463906 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:50.464116 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:50.468312 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:50.468326 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:50.468350 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:50.468411 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:50.472414 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:50.473348 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:50.473359 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:50.473368 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:50.473620 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:50.474007 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:50.474013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:50.474016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:50.474855 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:50.475335 (+ 480us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:50.475987 (+ 652us) server_negotiation.cc:299] Negotiation successful
0929 11:26:50.476631 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":353,"threads_started":1}
I20260929 11:26:50.483625 27298 raft_consensus.cc:348] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40ee61a2c5148db8444ceaba60b0544" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41449 } }
I20260929 11:26:50.484045 27298 raft_consensus.cc:374] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:50.484124 27298 raft_consensus.cc:729] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e40ee61a2c5148db8444ceaba60b0544, State: Initialized, Role: FOLLOWER
I20260929 11:26:50.484665 27298 consensus_queue.cc:260] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40ee61a2c5148db8444ceaba60b0544" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41449 } }
I20260929 11:26:50.484874 27298 raft_consensus.cc:388] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:26:50.484988 27298 raft_consensus.cc:482] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:26:50.485117 27298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:50.487366 27298 raft_consensus.cc:504] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40ee61a2c5148db8444ceaba60b0544" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41449 } }
I20260929 11:26:50.487948 27298 leader_election.cc:302] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e40ee61a2c5148db8444ceaba60b0544; no voters:
I20260929 11:26:50.488579 27298 leader_election.cc:288] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:26:50.488716 27303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:50.489326 27303 raft_consensus.cc:686] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544 [term 1 LEADER]: Becoming Leader. State: Replica: e40ee61a2c5148db8444ceaba60b0544, State: Running, Role: LEADER
I20260929 11:26:50.489857 27303 consensus_queue.cc:237] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40ee61a2c5148db8444ceaba60b0544" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41449 } }
I20260929 11:26:50.490603 27298 sys_catalog.cc:564] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:26:50.492667 27305 sys_catalog.cc:455] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e40ee61a2c5148db8444ceaba60b0544. Latest consensus state: current_term: 1 leader_uuid: "e40ee61a2c5148db8444ceaba60b0544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40ee61a2c5148db8444ceaba60b0544" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41449 } } }
I20260929 11:26:50.493700 27305 sys_catalog.cc:458] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:50.492875 27304 sys_catalog.cc:455] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e40ee61a2c5148db8444ceaba60b0544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40ee61a2c5148db8444ceaba60b0544" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41449 } } }
I20260929 11:26:50.494143 27304 sys_catalog.cc:458] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:50.496343 27313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:26:50.500716 27313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:26:50.510120 27313 catalog_manager.cc:1347] Generated new cluster ID: 8ab8e4e4a44a4baf95fde31f94ab6060
I20260929 11:26:50.510272 27313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:26:50.524219 27313 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:26:50.526366 27313 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:26:50.540429 27313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e40ee61a2c5148db8444ceaba60b0544: Generated new TSK 0
I20260929 11:26:50.541954 27313 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
I20260929 11:26:50.589366 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:41449
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:50.951704 27324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:50.952036 27324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:50.952181 27324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:50.960035 27324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:50.960192 27324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:50.960354 27324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:26:50.970554 27324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:41449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:50.972046 27324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:50.973448 27324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:50.987255 27330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:50.987373 27329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:50.987317 27332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:50.988682 27324 server_base.cc:1034] running on GCE node
I20260929 11:26:50.989603 27324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:50.990823 27324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:50.992071 27324 hybrid_clock.cc:648] HybridClock initialized: now 1790681210991965 us; error 108 us; skew 500 ppm
I20260929 11:26:50.992424 27324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:50 dist-test-slave-qk05 krb5kdc[27218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681210, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:50.996742 27324 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:26:50.998623 27324 webserver.cc:460] Webserver started at http://127.23.153.1:37575/ using document root <none> and password file <none>
I20260929 11:26:50.999429 27324 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:50.999562 27324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:50.999857 27324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:51.003196 27324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "d95a5a41706842e7bff85403375f95b5"
format_stamp: "Formatted at 2026-09-29 11:26:50 on dist-test-slave-qk05"
server_key: "76b9c9e633d884c2dac29a8156d50285"
server_key_iv: "c1417396bb4a10c5a4e64206ba0c41bc"
server_key_version: "encryptionkey@0"
I20260929 11:26:51.004037 27324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "d95a5a41706842e7bff85403375f95b5"
format_stamp: "Formatted at 2026-09-29 11:26:50 on dist-test-slave-qk05"
server_key: "76b9c9e633d884c2dac29a8156d50285"
server_key_iv: "c1417396bb4a10c5a4e64206ba0c41bc"
server_key_version: "encryptionkey@0"
I20260929 11:26:51.011269 27324 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 11:26:51.016783 27339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:51.018589 27324 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:26:51.018805 27324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "d95a5a41706842e7bff85403375f95b5"
format_stamp: "Formatted at 2026-09-29 11:26:50 on dist-test-slave-qk05"
server_key: "76b9c9e633d884c2dac29a8156d50285"
server_key_iv: "c1417396bb4a10c5a4e64206ba0c41bc"
server_key_version: "encryptionkey@0"
I20260929 11:26:51.019093 27324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:51.049598 27324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:51.054454 27324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:51.054793 27324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:51.055882 27324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:26:51.057291 27324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:51.057372 27324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:51.057493 27324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:51.057545 27324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:51.091472 27324 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:41799
I20260929 11:26:51.091529 27451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:41799 every 8 connection(s)
I20260929 11:26:51.092998 27324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:26:51.102123 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 27324
I20260929 11:26:51.102316 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:26:51.102948 24164 external_mini_cluster.cc:1428] Setting key 5c93e3cc19f2aee8f0e8b0ab7cff28af
Sep 29 11:26:51 dist-test-slave-qk05 krb5kdc[27218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681210, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:51.116932 27454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:51.095933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:54199 (local address 127.23.153.62:41449)
0929 11:26:51.096470 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:51.096476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:51.097640 (+ 1164us) server_negotiation.cc:407] Connection header received
0929 11:26:51.099647 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:51.099653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:51.099754 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:51.099847 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:51.101576 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:51.102338 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:51.104101 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:51.104524 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:51.104652 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:51.109205 (+ 4553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:51.109230 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:51.109236 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:51.109306 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:51.112238 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:51.113256 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:51.113265 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:51.113268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:51.113508 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:51.114235 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:51.114239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:51.114242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:51.114591 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:51.114753 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:51.116208 (+ 1455us) server_negotiation.cc:299] Negotiation successful
0929 11:26:51.116561 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":314,"threads_started":1}
I20260929 11:26:51.117848 27455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:51.096630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41449 (local address 127.23.153.1:54199)
0929 11:26:51.097469 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:51.097511 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:51.099330 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:51.100129 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:51.100139 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:51.100746 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:51.101366 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:51.101384 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:51.102472 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:51.102477 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:51.103948 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:51.103970 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:51.104746 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:51.104752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:51.105068 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:51.105846 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:51.105874 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:51.108781 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:51.112450 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:51.112461 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:51.112478 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:51.112998 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:51.113755 (+ 757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:51.113760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:51.113763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:51.114085 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:51.114749 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:51.114761 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:51.115316 (+ 555us) client_negotiation.cc:769] Sending connection context
0929 11:26:51.115886 (+ 570us) client_negotiation.cc:240] Negotiation successful
0929 11:26:51.116573 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":287,"threads_started":1}
I20260929 11:26:51.119658 27452 heartbeater.cc:347] Connected to a master server at 127.23.153.62:41449
I20260929 11:26:51.119992 27452 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.153.2@KRBTEST.COM; defaulting to no policy
I20260929 11:26:51.120802 27452 heartbeater.cc:511] Master 127.23.153.62:41449 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.2@KRBTEST.COM" created.
I20260929 11:26:51.123876 27263 ts_manager.cc:194] Registered new tserver with Master: d95a5a41706842e7bff85403375f95b5 (127.23.153.1:41799)
I20260929 11:26:51.125751 27263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:54199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
I20260929 11:26:51.147692 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:41449
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:51.497802 27458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:51.498108 27458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:51.498206 27458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:51.506716 27458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:51.506865 27458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:51.507059 27458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:26:51.517874 27458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:41449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:51.519315 27458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:51.520723 27458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:51.535261 27463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:51.535413 27464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:51.535446 27466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:51.536028 27458 server_base.cc:1034] running on GCE node
I20260929 11:26:51.536737 27458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:51.537856 27458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:51.539121 27458 hybrid_clock.cc:648] HybridClock initialized: now 1790681211539034 us; error 71 us; skew 500 ppm
I20260929 11:26:51.539476 27458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:51 dist-test-slave-qk05 krb5kdc[27218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681211, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:51.542706 27458 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:26:51.544341 27458 webserver.cc:460] Webserver started at http://127.23.153.2:40183/ using document root <none> and password file <none>
I20260929 11:26:51.545058 27458 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:51.545168 27458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:51.545454 27458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:51.548697 27458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "2b1de177fca44b54bb06651b1b91e40a"
format_stamp: "Formatted at 2026-09-29 11:26:51 on dist-test-slave-qk05"
server_key: "3a8ac8affaf04008fdd37c4632623fe2"
server_key_iv: "ac223da87d10e1326aefaee8e54e05be"
server_key_version: "encryptionkey@0"
I20260929 11:26:51.549525 27458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "2b1de177fca44b54bb06651b1b91e40a"
format_stamp: "Formatted at 2026-09-29 11:26:51 on dist-test-slave-qk05"
server_key: "3a8ac8affaf04008fdd37c4632623fe2"
server_key_iv: "ac223da87d10e1326aefaee8e54e05be"
server_key_version: "encryptionkey@0"
I20260929 11:26:51.555581 27458 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 11:26:51.559634 27473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:51.561228 27458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:26:51.561429 27458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "2b1de177fca44b54bb06651b1b91e40a"
format_stamp: "Formatted at 2026-09-29 11:26:51 on dist-test-slave-qk05"
server_key: "3a8ac8affaf04008fdd37c4632623fe2"
server_key_iv: "ac223da87d10e1326aefaee8e54e05be"
server_key_version: "encryptionkey@0"
I20260929 11:26:51.561648 27458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:51.584576 27458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:51.589131 27458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:51.589440 27458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:51.590530 27458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:26:51.592135 27458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:51.592221 27458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:51.592326 27458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:51.592381 27458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:51.629276 27458 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:34015
I20260929 11:26:51.629304 27585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:34015 every 8 connection(s)
I20260929 11:26:51.630638 27458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:26:51.637874 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 27458
I20260929 11:26:51.638118 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:26:51.638713 24164 external_mini_cluster.cc:1428] Setting key 10a0e285d0da6a22d7f9566c184815c8
Sep 29 11:26:51 dist-test-slave-qk05 krb5kdc[27218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681211, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:51.658440 27588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:51.633705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:54385 (local address 127.23.153.62:41449)
0929 11:26:51.634214 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:51.634219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:51.635483 (+ 1264us) server_negotiation.cc:407] Connection header received
0929 11:26:51.639007 (+ 3524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:51.639035 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:51.639131 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:51.639235 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:51.642426 (+ 3191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:51.643283 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:51.645939 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:51.646644 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:51.646867 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:51.651091 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:51.651100 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:51.651104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:51.651151 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:51.654314 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:51.655386 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:51.655393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:51.655397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:51.655577 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:51.656047 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:51.656053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:51.656057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:51.656576 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:51.656746 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:51.657756 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0929 11:26:51.658012 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":248,"threads_started":1}
WARNING: no policy specified for kudu/127.23.153.3@KRBTEST.COM; defaulting to no policy
I20260929 11:26:51.659296 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:51.634278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41449 (local address 127.23.153.2:54385)
0929 11:26:51.635053 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:51.635241 (+ 188us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:51.638553 (+ 3312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:51.639764 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:51.639778 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:51.640658 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:51.642016 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:51.642075 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:51.643473 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:51.643479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:51.645697 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:51.645722 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:51.646998 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:51.647007 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:51.647394 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:51.648209 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:51.648240 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:51.650736 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:51.654537 (+ 3801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:51.654547 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:51.654566 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:51.655242 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:51.655702 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:51.655706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:51.655708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:51.655918 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:51.656755 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:51.656762 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:51.657321 (+ 559us) client_negotiation.cc:769] Sending connection context
0929 11:26:51.657589 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 11:26:51.658025 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":287,"threads_started":1}
I20260929 11:26:51.661007 27586 heartbeater.cc:347] Connected to a master server at 127.23.153.62:41449
I20260929 11:26:51.661358 27586 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.3@KRBTEST.COM" created.
I20260929 11:26:51.662312 27586 heartbeater.cc:511] Master 127.23.153.62:41449 requested a full tablet report, sending...
I20260929 11:26:51.664166 27263 ts_manager.cc:194] Registered new tserver with Master: 2b1de177fca44b54bb06651b1b91e40a (127.23.153.2:34015)
I20260929 11:26:51.665025 27263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:54385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
I20260929 11:26:51.685822 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:41449
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:52.050125 27592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:52.050451 27592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:52.050551 27592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:52.058434 27592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:52.058573 27592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:52.058722 27592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:26:52.071056 27592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:41449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:52.072482 27592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:52.074591 27592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:52.093662 27598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:52.093647 27597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:52.096489 27600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:52.099874 27592 server_base.cc:1034] running on GCE node
I20260929 11:26:52.101087 27592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:52.102340 27592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:52.103590 27592 hybrid_clock.cc:648] HybridClock initialized: now 1790681212103537 us; error 46 us; skew 500 ppm
I20260929 11:26:52.103940 27592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:52 dist-test-slave-qk05 krb5kdc[27218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681212, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:52.107823 27592 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:26:52.109977 27592 webserver.cc:460] Webserver started at http://127.23.153.3:33125/ using document root <none> and password file <none>
I20260929 11:26:52.111282 27592 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:52.111434 27592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:52.111733 27592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:52.115070 27592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "23503044cb8642e6b103510fdc279953"
format_stamp: "Formatted at 2026-09-29 11:26:52 on dist-test-slave-qk05"
server_key: "0f11bf311f19481e7dee51396d43ee1a"
server_key_iv: "b17e46a81213239ce2daaf09b2bdd20a"
server_key_version: "encryptionkey@0"
I20260929 11:26:52.115962 27592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "23503044cb8642e6b103510fdc279953"
format_stamp: "Formatted at 2026-09-29 11:26:52 on dist-test-slave-qk05"
server_key: "0f11bf311f19481e7dee51396d43ee1a"
server_key_iv: "b17e46a81213239ce2daaf09b2bdd20a"
server_key_version: "encryptionkey@0"
I20260929 11:26:52.122272 27592 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260929 11:26:52.126642 27607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:52.128403 27592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:26:52.128611 27592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "23503044cb8642e6b103510fdc279953"
format_stamp: "Formatted at 2026-09-29 11:26:52 on dist-test-slave-qk05"
server_key: "0f11bf311f19481e7dee51396d43ee1a"
server_key_iv: "b17e46a81213239ce2daaf09b2bdd20a"
server_key_version: "encryptionkey@0"
I20260929 11:26:52.128834 27592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:52.129683 27452 heartbeater.cc:503] Master 127.23.153.62:41449 was elected leader, sending a full tablet report...
I20260929 11:26:52.155932 27592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:52.161324 27592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:52.161629 27592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:52.162626 27592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:26:52.164127 27592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:52.164209 27592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:52.164327 27592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:52.164389 27592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:52.196874 27592 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:39443
I20260929 11:26:52.196926 27719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:39443 every 8 connection(s)
I20260929 11:26:52.198282 27592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:26:52.208325 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 27592
I20260929 11:26:52.208509 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:26:52.208943 24164 external_mini_cluster.cc:1428] Setting key 253b951b3533623457c47b134769c430
Sep 29 11:26:52 dist-test-slave-qk05 krb5kdc[27218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681212, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:52.221836 27722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:52.201065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:59593 (local address 127.23.153.62:41449)
0929 11:26:52.201563 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:52.201570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:52.202782 (+ 1212us) server_negotiation.cc:407] Connection header received
0929 11:26:52.205271 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:52.205276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:52.205358 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:52.205466 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:52.207445 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:52.208290 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:52.210258 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:52.210730 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:52.210873 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:52.214922 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:52.214928 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:52.214931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:52.214969 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:52.217871 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:52.218866 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:52.218871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:52.218873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:52.219080 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:52.219599 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:52.219603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:52.219606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:52.219897 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:52.220042 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:52.221325 (+ 1283us) server_negotiation.cc:299] Negotiation successful
0929 11:26:52.221547 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":221,"threads_started":1}
I20260929 11:26:52.223491 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:52.201637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41449 (local address 127.23.153.3:59593)
0929 11:26:52.202587 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:52.202633 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:52.204904 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:52.205782 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:52.205792 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:52.206397 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:52.207219 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:52.207248 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:52.208501 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:52.208507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:52.210082 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:52.210100 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:52.211138 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:52.211152 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:52.211448 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:52.212261 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:52.212295 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:52.214648 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:52.218144 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:52.218160 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:52.218178 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:52.218696 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:52.219222 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:52.219227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:52.219230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:52.219451 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:52.220160 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:52.220188 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:52.220851 (+ 663us) client_negotiation.cc:769] Sending connection context
0929 11:26:52.221175 (+ 324us) client_negotiation.cc:240] Negotiation successful
0929 11:26:52.221669 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":419,"threads_started":1}
I20260929 11:26:52.225207 27720 heartbeater.cc:347] Connected to a master server at 127.23.153.62:41449
I20260929 11:26:52.225529 27720 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:52.226320 27720 heartbeater.cc:511] Master 127.23.153.62:41449 requested a full tablet report, sending...
I20260929 11:26:52.228029 27262 ts_manager.cc:194] Registered new tserver with Master: 23503044cb8642e6b103510fdc279953 (127.23.153.3:39443)
I20260929 11:26:52.229041 27262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:59593
I20260929 11:26:52.237751 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 11:26:52 dist-test-slave-qk05 krb5kdc[27218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681212, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 11:26:52 dist-test-slave-qk05 krb5kdc[27218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681212, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:52.286897 27722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:52.272179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37904 (local address 127.23.153.62:41449)
0929 11:26:52.272424 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:52.272430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:52.272740 (+ 310us) server_negotiation.cc:407] Connection header received
0929 11:26:52.272908 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:52.272912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:52.272984 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:52.273083 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:52.274554 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:52.275389 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:52.276759 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:52.277230 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:52.277340 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:52.281383 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:52.281390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:52.281393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:52.281446 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:52.284107 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:52.284802 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:52.284805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:52.284808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:52.284929 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:52.285371 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:52.285376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:52.285378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:52.285754 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:52.285948 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:52.286334 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 11:26:52.286569 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260929 11:26:52.287457 27262 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37904
I20260929 11:26:52.289402 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 27324
Sep 29 11:26:52 dist-test-slave-qk05 krb5kdc[27218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681212, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:26:52.307438 27732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:52.290495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52418 (local address 127.23.153.1:41799)
0929 11:26:52.291118 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:52.291125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:52.291168 (+ 43us) server_negotiation.cc:407] Connection header received
0929 11:26:52.291226 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:52.291233 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:52.291443 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:52.291589 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:52.292763 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:52.293836 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:52.295211 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:52.295808 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:52.295915 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:52.301004 (+ 5089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:52.301023 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:52.301041 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:52.301180 (+ 139us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:52.304313 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:52.305316 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:52.305325 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:52.305328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:52.305501 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:52.305952 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:52.305955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:52.305957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:52.306348 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:52.306595 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:52.306829 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 11:26:52.307142 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":290,"threads_started":1}
W20260929 11:26:52.308729 27447 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52418
W20260929 11:26:52.309976 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 27324: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:26:52.310225 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 27324
I20260929 11:26:52.324608 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 27458
Sep 29 11:26:52 dist-test-slave-qk05 krb5kdc[27218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681212, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.153.2@KRBTEST.COM
I20260929 11:26:52.343446 27733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:52.325719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.23.153.2:34015)
0929 11:26:52.326428 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:52.326436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:52.326461 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:26:52.326524 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:52.326531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:52.326745 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:52.326886 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:52.328361 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:52.329437 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:52.330861 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:52.331947 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:52.332196 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:52.337448 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:52.337467 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:52.337484 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:52.337536 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:52.340431 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:52.341205 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:52.341212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:52.341214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:52.341355 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:52.341741 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:52.341745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:52.341747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:52.342122 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:52.342433 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:52.342756 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 11:26:52.343074 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":382,"threads_started":1}
W20260929 11:26:52.344839 27581 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54746
W20260929 11:26:52.346258 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 27458: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:26:52.346434 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 27458
I20260929 11:26:52.359064 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 27592
Sep 29 11:26:52 dist-test-slave-qk05 krb5kdc[27218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681212, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.153.3@KRBTEST.COM
I20260929 11:26:52.377203 27734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:52.360024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.23.153.3:39443)
0929 11:26:52.360700 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:52.360707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:52.360736 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:26:52.360795 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:52.360802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:52.361017 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:52.361201 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:52.362493 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:52.363634 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:52.364917 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:52.365509 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:52.365645 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:52.371292 (+ 5647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:52.371308 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:52.371325 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:52.371374 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:52.374312 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:52.375257 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:52.375265 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:52.375268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:52.375415 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:52.375718 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:52.375721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:52.375723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:52.376081 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:52.376327 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:52.376636 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 11:26:52.376861 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":373,"threads_started":1}
W20260929 11:26:52.378463 27715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42834
W20260929 11:26:52.379664 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 27592: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:26:52.379837 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 27592
I20260929 11:26:52.392292 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 27231
I20260929 11:26:52.393539 27293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:52.498960 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 27231
2026-09-29T11:26:52Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3745 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:26:52 dist-test-slave-qk05 krb5kdc[27738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:26:52 dist-test-slave-qk05 krb5kdc[27738](info): set up 2 sockets
Sep 29 11:26:52 dist-test-slave-qk05 krb5kdc[27738](info): commencing operation
krb5kdc: starting...
W20260929 11:26:53.527467 24164 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:26:53 dist-test-slave-qk05 krb5kdc[27738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:26:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:26:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:26:53.742341 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:40615
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:40615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:54.103222 27751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:54.103523 27751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:54.103659 27751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:54.111163 27751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:54.111330 27751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:54.111414 27751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:54.111485 27751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:54.111541 27751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:54.122205 27751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:40615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:40615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:54.123870 27751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:54.125540 27751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:54.137909 27756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:54.137892 27757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:54.138073 27759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:54.138679 27751 server_base.cc:1034] running on GCE node
I20260929 11:26:54.139611 27751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:54.141186 27751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:54.142431 27751 hybrid_clock.cc:648] HybridClock initialized: now 1790681214142383 us; error 44 us; skew 500 ppm
I20260929 11:26:54.142819 27751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:54 dist-test-slave-qk05 krb5kdc[27738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681214, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:54.146529 27751 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:26:54.148205 27751 webserver.cc:460] Webserver started at http://127.23.153.62:33129/ using document root <none> and password file <none>
I20260929 11:26:54.148978 27751 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:54.149107 27751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:54.149434 27751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:54.152817 27751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "57b5517078b546959d7ed011cf6ce746"
format_stamp: "Formatted at 2026-09-29 11:26:54 on dist-test-slave-qk05"
server_key: "d5fc039ab237f2abfb8ad097fd2aad2a"
server_key_iv: "510ca29d06d3f2dbad3d041c220ab4cc"
server_key_version: "encryptionkey@0"
I20260929 11:26:54.153712 27751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "57b5517078b546959d7ed011cf6ce746"
format_stamp: "Formatted at 2026-09-29 11:26:54 on dist-test-slave-qk05"
server_key: "d5fc039ab237f2abfb8ad097fd2aad2a"
server_key_iv: "510ca29d06d3f2dbad3d041c220ab4cc"
server_key_version: "encryptionkey@0"
I20260929 11:26:54.159427 27751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 11:26:54.163245 27766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:54.164686 27751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:26:54.164873 27751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "57b5517078b546959d7ed011cf6ce746"
format_stamp: "Formatted at 2026-09-29 11:26:54 on dist-test-slave-qk05"
server_key: "d5fc039ab237f2abfb8ad097fd2aad2a"
server_key_iv: "510ca29d06d3f2dbad3d041c220ab4cc"
server_key_version: "encryptionkey@0"
I20260929 11:26:54.165077 27751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:54.207849 27751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:54.212656 27751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:54.212965 27751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:54.234503 27751 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:40615
I20260929 11:26:54.234601 27817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:40615 every 8 connection(s)
I20260929 11:26:54.236007 27751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:26:54.240222 27818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:26:54.244388 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 27751
I20260929 11:26:54.244626 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:26:54.245404 24164 external_mini_cluster.cc:1428] Setting key ffd629b0981dd881d1a0fabdd7008700
I20260929 11:26:54.255048 27818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746: Bootstrap starting.
I20260929 11:26:54.259703 27818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:54.261504 27818 log.cc:824] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746: Log is configured to *not* fsync() on all Append() calls
Sep 29 11:26:54 dist-test-slave-qk05 krb5kdc[27738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:54.264989 27818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746: No bootstrap required, opened a new log
I20260929 11:26:54.271628 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:54.247872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41790 (local address 127.23.153.62:40615)
0929 11:26:54.248679 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:54.248693 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:54.248738 (+ 45us) server_negotiation.cc:407] Connection header received
0929 11:26:54.250432 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:54.250461 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:54.250951 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:54.251385 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:54.253313 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:54.254738 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:54.256598 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:54.257506 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:54.257668 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:54.263468 (+ 5800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:54.263478 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:54.263497 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:54.263542 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:54.266827 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:54.267695 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:54.267703 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:54.267714 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:54.267925 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:54.268325 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:54.268329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:54.268331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:54.269013 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:54.269337 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:54.269856 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 11:26:54.270359 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":375,"threads_started":1}
I20260929 11:26:54.273612 27818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b5517078b546959d7ed011cf6ce746" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40615 } }
I20260929 11:26:54.274008 27818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:54.274106 27818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b5517078b546959d7ed011cf6ce746, State: Initialized, Role: FOLLOWER
I20260929 11:26:54.274741 27818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b5517078b546959d7ed011cf6ce746" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40615 } }
I20260929 11:26:54.275060 27818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:26:54.275194 27818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:26:54.275341 27818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:54.277549 27818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b5517078b546959d7ed011cf6ce746" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40615 } }
I20260929 11:26:54.278095 27818 leader_election.cc:302] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57b5517078b546959d7ed011cf6ce746; no voters:
I20260929 11:26:54.278813 27818 leader_election.cc:288] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:26:54.278942 27823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:54.280308 27823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746 [term 1 LEADER]: Becoming Leader. State: Replica: 57b5517078b546959d7ed011cf6ce746, State: Running, Role: LEADER
I20260929 11:26:54.280879 27823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b5517078b546959d7ed011cf6ce746" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40615 } }
I20260929 11:26:54.281230 27818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:26:54.283501 27825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57b5517078b546959d7ed011cf6ce746. Latest consensus state: current_term: 1 leader_uuid: "57b5517078b546959d7ed011cf6ce746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b5517078b546959d7ed011cf6ce746" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40615 } } }
I20260929 11:26:54.283890 27825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:54.284322 27824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57b5517078b546959d7ed011cf6ce746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b5517078b546959d7ed011cf6ce746" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40615 } } }
I20260929 11:26:54.284621 27824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:54.286985 27829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:26:54.291963 27829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:26:54.301942 27829 catalog_manager.cc:1347] Generated new cluster ID: 8bdd35300a8d4f8d82f7e783f0d8ff65
I20260929 11:26:54.302083 27829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:26:54.321512 27829 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:26:54.322942 27829 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:26:54.332448 27829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57b5517078b546959d7ed011cf6ce746: Generated new TSK 0
I20260929 11:26:54.333657 27829 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
I20260929 11:26:54.384737 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40615
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:54.742094 27844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:54.742404 27844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:54.742499 27844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:54.752233 27844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:54.752385 27844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:54.752508 27844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:26:54.763374 27844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:54.765266 27844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:54.767114 27844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:54.782924 27849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:54.783289 27850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:54.783381 27852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:54.784269 27844 server_base.cc:1034] running on GCE node
I20260929 11:26:54.784966 27844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:54.786254 27844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:54.787508 27844 hybrid_clock.cc:648] HybridClock initialized: now 1790681214787453 us; error 51 us; skew 500 ppm
I20260929 11:26:54.787878 27844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:54 dist-test-slave-qk05 krb5kdc[27738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681214, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:54.791352 27844 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:26:54.792963 27844 webserver.cc:460] Webserver started at http://127.23.153.1:40249/ using document root <none> and password file <none>
I20260929 11:26:54.793710 27844 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:54.793835 27844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:54.794147 27844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:54.797403 27844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "9b589c9ddcf4483197566560f6795bcb"
format_stamp: "Formatted at 2026-09-29 11:26:54 on dist-test-slave-qk05"
server_key: "206f68477fb1ac48e0cce5844f1017fc"
server_key_iv: "5d270f1aba9be0955bf74c148110869a"
server_key_version: "encryptionkey@0"
I20260929 11:26:54.798317 27844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "9b589c9ddcf4483197566560f6795bcb"
format_stamp: "Formatted at 2026-09-29 11:26:54 on dist-test-slave-qk05"
server_key: "206f68477fb1ac48e0cce5844f1017fc"
server_key_iv: "5d270f1aba9be0955bf74c148110869a"
server_key_version: "encryptionkey@0"
I20260929 11:26:54.804837 27844 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 11:26:54.809481 27859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:54.811111 27844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:26:54.811298 27844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "9b589c9ddcf4483197566560f6795bcb"
format_stamp: "Formatted at 2026-09-29 11:26:54 on dist-test-slave-qk05"
server_key: "206f68477fb1ac48e0cce5844f1017fc"
server_key_iv: "5d270f1aba9be0955bf74c148110869a"
server_key_version: "encryptionkey@0"
I20260929 11:26:54.811510 27844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:54.838176 27844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:54.843083 27844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:54.843364 27844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:54.844393 27844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:26:54.845965 27844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:54.846051 27844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:54.846154 27844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:54.846211 27844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:54.877512 27844 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:44115
I20260929 11:26:54.877555 27971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:44115 every 8 connection(s)
I20260929 11:26:54.878816 27844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:26:54.885949 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 27844
I20260929 11:26:54.886174 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:26:54.886817 24164 external_mini_cluster.cc:1428] Setting key 0a45426d559b8662cae6cfae653a3dd6
Sep 29 11:26:54 dist-test-slave-qk05 krb5kdc[27738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681214, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:54.901751 27974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:54.881720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:39835 (local address 127.23.153.62:40615)
0929 11:26:54.882199 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:54.882204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:54.883316 (+ 1112us) server_negotiation.cc:407] Connection header received
0929 11:26:54.885252 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:54.885258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:54.885357 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:54.885468 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:54.887396 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:54.888210 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:54.889943 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:54.890361 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:54.890479 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:54.894898 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:54.894907 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:54.894912 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:54.894958 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:54.898256 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:54.899225 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:54.899231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:54.899234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:54.899388 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:54.899882 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:54.899887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:54.899889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:54.900226 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:54.900373 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:54.901161 (+ 788us) server_negotiation.cc:299] Negotiation successful
0929 11:26:54.901411 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":272,"threads_started":1}
I20260929 11:26:54.902624 27975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:54.882319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:40615 (local address 127.23.153.1:39835)
0929 11:26:54.883147 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:54.883191 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:54.884927 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:54.885774 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:54.885783 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:54.886478 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:54.887186 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:54.887209 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:54.888418 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:54.888422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:54.889757 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:54.889778 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:54.890678 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:54.890685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:54.891053 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:54.891804 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:54.891833 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:54.894567 (+ 2734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:54.898475 (+ 3908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:54.898485 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:54.898502 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:54.899072 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:54.899532 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:54.899536 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:54.899539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:54.899765 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:54.900387 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:54.900395 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:54.900761 (+ 366us) client_negotiation.cc:769] Sending connection context
0929 11:26:54.901020 (+ 259us) client_negotiation.cc:240] Negotiation successful
0929 11:26:54.901453 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":305,"threads_started":1}
I20260929 11:26:54.904439 27972 heartbeater.cc:347] Connected to a master server at 127.23.153.62:40615
I20260929 11:26:54.904827 27972 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:54.905851 27972 heartbeater.cc:511] Master 127.23.153.62:40615 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.2@KRBTEST.COM" created.
I20260929 11:26:54.908981 27783 ts_manager.cc:194] Registered new tserver with Master: 9b589c9ddcf4483197566560f6795bcb (127.23.153.1:44115)
I20260929 11:26:54.910583 27783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:39835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
I20260929 11:26:54.932586 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40615
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:55.293133 27978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:55.293460 27978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:55.293607 27978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:55.301195 27978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:55.301326 27978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:55.301481 27978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:26:55.312630 27978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:55.314190 27978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:55.315771 27978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:55.330433 27986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:55.330463 27984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:55.330878 27983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:55.331589 27978 server_base.cc:1034] running on GCE node
I20260929 11:26:55.332283 27978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:55.333436 27978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:55.334684 27978 hybrid_clock.cc:648] HybridClock initialized: now 1790681215334627 us; error 60 us; skew 500 ppm
I20260929 11:26:55.335117 27978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:55 dist-test-slave-qk05 krb5kdc[27738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681215, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:55.338610 27978 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:26:55.340409 27978 webserver.cc:460] Webserver started at http://127.23.153.2:33151/ using document root <none> and password file <none>
I20260929 11:26:55.341166 27978 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:55.341338 27978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:55.341727 27978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:55.345032 27978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "ebf5a38491c44978862f6b0cdce1c7d7"
format_stamp: "Formatted at 2026-09-29 11:26:55 on dist-test-slave-qk05"
server_key: "60831d7d7d6449921fdc8a98b437b0fb"
server_key_iv: "51fe0bbbc6f61c454cc42872be2003b2"
server_key_version: "encryptionkey@0"
I20260929 11:26:55.345984 27978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "ebf5a38491c44978862f6b0cdce1c7d7"
format_stamp: "Formatted at 2026-09-29 11:26:55 on dist-test-slave-qk05"
server_key: "60831d7d7d6449921fdc8a98b437b0fb"
server_key_iv: "51fe0bbbc6f61c454cc42872be2003b2"
server_key_version: "encryptionkey@0"
I20260929 11:26:55.351964 27978 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 11:26:55.355842 27993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:55.357404 27978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:26:55.357590 27978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "ebf5a38491c44978862f6b0cdce1c7d7"
format_stamp: "Formatted at 2026-09-29 11:26:55 on dist-test-slave-qk05"
server_key: "60831d7d7d6449921fdc8a98b437b0fb"
server_key_iv: "51fe0bbbc6f61c454cc42872be2003b2"
server_key_version: "encryptionkey@0"
I20260929 11:26:55.357821 27978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:55.401976 27978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:55.408004 27978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:55.408339 27978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:55.409415 27978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:26:55.411083 27978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:55.411181 27978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 11:26:55.411312 27978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:55.411365 27978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:55.446116 27978 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:43839
I20260929 11:26:55.446180 28105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:43839 every 8 connection(s)
I20260929 11:26:55.447475 27978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:26:55.453738 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 27978
I20260929 11:26:55.453938 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:26:55.454473 24164 external_mini_cluster.cc:1428] Setting key 4aa93757574e63b835f6a0b29e1d9ad1
Sep 29 11:26:55 dist-test-slave-qk05 krb5kdc[27738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681215, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:55.471562 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:55.450411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:43285 (local address 127.23.153.62:40615)
0929 11:26:55.450896 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:55.450902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:55.451924 (+ 1022us) server_negotiation.cc:407] Connection header received
0929 11:26:55.453883 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:55.453889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:55.453983 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:55.454091 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:55.456072 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:55.456951 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:55.459407 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:55.459854 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:55.459967 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:55.464428 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:55.464436 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:55.464442 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:55.464486 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:55.467715 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:55.468706 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:55.468713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:55.468716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:55.468891 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:55.469450 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:55.469455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:55.469459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:55.469829 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:55.469981 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:55.470856 (+ 875us) server_negotiation.cc:299] Negotiation successful
0929 11:26:55.471143 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":270,"threads_started":1}
I20260929 11:26:55.472409 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:55.450934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:40615 (local address 127.23.153.2:43285)
0929 11:26:55.451742 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:55.451784 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:55.453540 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:55.454427 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:55.454443 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:55.455243 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:55.455894 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:55.455915 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:55.457136 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:55.457141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:55.459197 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:55.459217 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:55.460097 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:55.460104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:55.460405 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:55.461163 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:55.461197 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:55.464085 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:55.467957 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:55.467968 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:55.467987 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:55.468540 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:55.469043 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:55.469047 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:55.469050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:55.469314 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:55.469991 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:55.469998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:55.470441 (+ 443us) client_negotiation.cc:769] Sending connection context
0929 11:26:55.470692 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 11:26:55.471179 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":312,"threads_started":1}
WARNING: no policy specified for kudu/127.23.153.3@KRBTEST.COM; defaulting to no policy
I20260929 11:26:55.474596 28106 heartbeater.cc:347] Connected to a master server at 127.23.153.62:40615
I20260929 11:26:55.475191 28106 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:55.476280 28106 heartbeater.cc:511] Master 127.23.153.62:40615 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.3@KRBTEST.COM" created.
I20260929 11:26:55.478044 27783 ts_manager.cc:194] Registered new tserver with Master: ebf5a38491c44978862f6b0cdce1c7d7 (127.23.153.2:43839)
I20260929 11:26:55.479110 27783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:43285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
I20260929 11:26:55.502168 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40615
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:55.852662 28112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:55.852989 28112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:55.853114 28112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:55.861055 28112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:55.861208 28112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:55.861368 28112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:26:55.872223 28112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:55.873579 28112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:55.874934 28112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:55.889587 28117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:55.889605 28118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:55.889871 28120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:55.890540 28112 server_base.cc:1034] running on GCE node
I20260929 11:26:55.891244 28112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:55.892287 28112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:55.893491 28112 hybrid_clock.cc:648] HybridClock initialized: now 1790681215893436 us; error 47 us; skew 500 ppm
I20260929 11:26:55.893816 28112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:55 dist-test-slave-qk05 krb5kdc[27738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681215, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:55.897053 28112 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:26:55.898679 28112 webserver.cc:460] Webserver started at http://127.23.153.3:40597/ using document root <none> and password file <none>
I20260929 11:26:55.899477 28112 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:55.899622 28112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:55.899941 28112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:55.903113 28112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "a351002200944eea8c20427beeb600ec"
format_stamp: "Formatted at 2026-09-29 11:26:55 on dist-test-slave-qk05"
server_key: "abc7096757825c6ded57d753a7bfc719"
server_key_iv: "df4704dc9f40588db77cec5f3bdbd75d"
server_key_version: "encryptionkey@0"
I20260929 11:26:55.904027 28112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "a351002200944eea8c20427beeb600ec"
format_stamp: "Formatted at 2026-09-29 11:26:55 on dist-test-slave-qk05"
server_key: "abc7096757825c6ded57d753a7bfc719"
server_key_iv: "df4704dc9f40588db77cec5f3bdbd75d"
server_key_version: "encryptionkey@0"
I20260929 11:26:55.910023 28112 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:26:55.914059 27972 heartbeater.cc:503] Master 127.23.153.62:40615 was elected leader, sending a full tablet report...
I20260929 11:26:55.914587 28127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:55.916558 28112 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 11:26:55.916730 28112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "a351002200944eea8c20427beeb600ec"
format_stamp: "Formatted at 2026-09-29 11:26:55 on dist-test-slave-qk05"
server_key: "abc7096757825c6ded57d753a7bfc719"
server_key_iv: "df4704dc9f40588db77cec5f3bdbd75d"
server_key_version: "encryptionkey@0"
I20260929 11:26:55.916934 28112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:55.940253 28112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:55.946058 28112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:55.946364 28112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:55.947700 28112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:26:55.949927 28112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:55.950037 28112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:55.950138 28112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:55.950181 28112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:55.981705 28112 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:45367
I20260929 11:26:55.981801 28239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:45367 every 8 connection(s)
I20260929 11:26:55.983074 28112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:26:55.992374 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 28112
I20260929 11:26:55.992563 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:26:55.993237 24164 external_mini_cluster.cc:1428] Setting key 81ed234d7da87647c77dfd798d95ed33
Sep 29 11:26:55 dist-test-slave-qk05 krb5kdc[27738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681215, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:56.005853 28242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:55.985838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:45857 (local address 127.23.153.62:40615)
0929 11:26:55.986350 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:55.986358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:55.987440 (+ 1082us) server_negotiation.cc:407] Connection header received
0929 11:26:55.989467 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:55.989472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:55.989558 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:55.989673 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:55.991627 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:55.992468 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:55.995035 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:55.995545 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:55.995685 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:55.999535 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:55.999541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:55.999544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:55.999612 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:56.002171 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:56.003144 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:56.003149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:56.003152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:56.003494 (+ 342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:56.004029 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:56.004033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:56.004036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:56.004354 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:56.004503 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:56.005336 (+ 833us) server_negotiation.cc:299] Negotiation successful
0929 11:26:56.005536 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":231,"threads_started":1}
I20260929 11:26:56.006855 28243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:55.986409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:40615 (local address 127.23.153.3:45857)
0929 11:26:55.987230 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:55.987276 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:55.989087 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:55.989990 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:55.990000 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:55.990584 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:55.991337 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:55.991366 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:55.992671 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:55.992677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:55.994847 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:55.994869 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:55.995794 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:55.995799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:55.996025 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:55.996738 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:55.996773 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:55.999253 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:56.002386 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:56.002395 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:56.002413 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:56.002944 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:56.003648 (+ 704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:56.003652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:56.003655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:56.003881 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:56.004523 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:56.004530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:56.004917 (+ 387us) client_negotiation.cc:769] Sending connection context
0929 11:26:56.005180 (+ 263us) client_negotiation.cc:240] Negotiation successful
0929 11:26:56.005638 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":296,"threads_started":1}
I20260929 11:26:56.008632 28240 heartbeater.cc:347] Connected to a master server at 127.23.153.62:40615
I20260929 11:26:56.009008 28240 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:56.009845 28240 heartbeater.cc:511] Master 127.23.153.62:40615 requested a full tablet report, sending...
I20260929 11:26:56.011436 27783 ts_manager.cc:194] Registered new tserver with Master: a351002200944eea8c20427beeb600ec (127.23.153.3:45367)
I20260929 11:26:56.012288 27783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:45857
I20260929 11:26:56.022827 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 11:26:56 dist-test-slave-qk05 krb5kdc[27738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 11:26:56 dist-test-slave-qk05 krb5kdc[27738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:26:56.077080 28248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:56.056554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.23.153.1:44115)
0929 11:26:56.057149 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:56.057156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:56.057185 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:26:56.057260 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:56.057267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:56.057499 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:56.057641 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:56.059166 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:56.060291 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:56.061665 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:56.062371 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:56.062525 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:56.070856 (+ 8331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:56.070874 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:56.070893 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:56.070944 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:56.073959 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:56.074802 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:56.074810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:56.074812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:56.074953 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:56.075321 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:56.075324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:56.075326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:56.075692 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:56.075919 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:56.076490 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 11:26:56.076786 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":295,"threads_started":1}
Sep 29 11:26:56 dist-test-slave-qk05 krb5kdc[27738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:56.097043 28242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:56.081753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41798 (local address 127.23.153.62:40615)
0929 11:26:56.082043 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:56.082049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:56.082282 (+ 233us) server_negotiation.cc:407] Connection header received
0929 11:26:56.082487 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:56.082491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:56.082558 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:56.082637 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:56.084036 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:56.084777 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:56.086270 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:56.086899 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:56.087333 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:56.091494 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:56.091500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:56.091504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:56.091548 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:56.094510 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:56.095346 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:56.095350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:56.095353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:56.095489 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:56.095812 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:56.095815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:56.095817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:56.096155 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:56.096334 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:56.096620 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 11:26:56.096816 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
W20260929 11:26:56.097666 27783 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41798
I20260929 11:26:56.098678 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 27844
I20260929 11:26:56.113348 28248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:56.100004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40116 (local address 127.23.153.1:44115)
0929 11:26:56.100227 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:56.100234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:56.100393 (+ 159us) server_negotiation.cc:407] Connection header received
0929 11:26:56.100509 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:56.100512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:56.100614 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:56.100704 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:56.102435 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:56.103195 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:56.104479 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:56.104940 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:56.105050 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:56.107759 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:56.107765 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:56.107769 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:56.107835 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:56.110549 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:56.111448 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:56.111452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:56.111455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:56.111606 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:56.111985 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:56.111989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:56.111992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:56.112283 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:56.112484 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:56.112844 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 11:26:56.113035 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 11:26:56.114202 27967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:56.199138 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 27844
I20260929 11:26:56.213204 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 27978
Sep 29 11:26:56 dist-test-slave-qk05 krb5kdc[27738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.2@KRBTEST.COM
I20260929 11:26:56.230749 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:56.214311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59962 (local address 127.23.153.2:43839)
0929 11:26:56.214940 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:56.214950 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:56.214980 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:26:56.215088 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:56.215097 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:56.215550 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:56.215794 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:56.217088 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:56.218147 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:56.219698 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:56.220308 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:56.220440 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:56.224726 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:56.224743 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:56.224761 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:56.224807 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:56.227866 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:56.228681 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:56.228689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:56.228692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:56.228834 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:56.229192 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:56.229196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:56.229198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:56.229584 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:56.229821 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:56.230120 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 11:26:56.230355 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":313,"threads_started":1}
I20260929 11:26:56.232101 28101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:56.320187 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 27978
I20260929 11:26:56.333263 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 28112
Sep 29 11:26:56 dist-test-slave-qk05 krb5kdc[27738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.3@KRBTEST.COM
I20260929 11:26:56.351904 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:56.334295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50088 (local address 127.23.153.3:45367)
0929 11:26:56.335076 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:56.335086 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:56.335126 (+ 40us) server_negotiation.cc:407] Connection header received
0929 11:26:56.335193 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:56.335200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:56.335443 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:56.335598 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:56.337086 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:56.338149 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:56.339569 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:56.340215 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:56.340357 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:56.345659 (+ 5302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:56.345677 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:56.345695 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:56.345752 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:56.348955 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:56.349782 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:56.349790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:56.349795 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:56.349942 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:56.350308 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:56.350313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:56.350315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:56.350712 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:56.350974 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:56.351342 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 11:26:56.351578 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":360,"threads_started":1}
I20260929 11:26:56.353327 28235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:56.439131 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 28112
I20260929 11:26:56.452334 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 27751
I20260929 11:26:56.453646 27813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:26:56.558945 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 27751
2026-09-29T11:26:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4057 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:26:56 dist-test-slave-qk05 krb5kdc[28260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:26:56 dist-test-slave-qk05 krb5kdc[28260](info): set up 2 sockets
Sep 29 11:26:56 dist-test-slave-qk05 krb5kdc[28260](info): commencing operation
krb5kdc: starting...
W20260929 11:26:57.579317 24164 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:26:57 dist-test-slave-qk05 krb5kdc[28260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:26:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:26:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:26:57.780409 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:40691
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:40691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:58.137342 28273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:58.137677 28273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:58.137796 28273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:58.145808 28273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:26:58.146157 28273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:58.146271 28273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:58.146337 28273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:26:58.146386 28273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:26:58.157459 28273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:40691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:40691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:58.158898 28273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:58.160308 28273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:58.172971 28278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:58.172916 28279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:58.173466 28281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:58.173625 28273 server_base.cc:1034] running on GCE node
I20260929 11:26:58.174520 28273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:58.176843 28273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:58.178094 28273 hybrid_clock.cc:648] HybridClock initialized: now 1790681218178050 us; error 48 us; skew 500 ppm
I20260929 11:26:58.178530 28273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:58 dist-test-slave-qk05 krb5kdc[28260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681218, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:58.182050 28273 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:26:58.183575 28273 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:58.183694 28273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:58.184024 28273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:58.187423 28273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "b3559328b684408c81618658de962b05"
format_stamp: "Formatted at 2026-09-29 11:26:58 on dist-test-slave-qk05"
server_key: "69a4a8a712a6553ff550f0d2ec0a6fb1"
server_key_iv: "9b83ba98021efb47a33bf97967ffbf08"
server_key_version: "encryptionkey@0"
I20260929 11:26:58.188347 28273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "b3559328b684408c81618658de962b05"
format_stamp: "Formatted at 2026-09-29 11:26:58 on dist-test-slave-qk05"
server_key: "69a4a8a712a6553ff550f0d2ec0a6fb1"
server_key_iv: "9b83ba98021efb47a33bf97967ffbf08"
server_key_version: "encryptionkey@0"
I20260929 11:26:58.194204 28273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:26:58.198350 28287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:58.200487 28273 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:26:58.200722 28273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "b3559328b684408c81618658de962b05"
format_stamp: "Formatted at 2026-09-29 11:26:58 on dist-test-slave-qk05"
server_key: "69a4a8a712a6553ff550f0d2ec0a6fb1"
server_key_iv: "9b83ba98021efb47a33bf97967ffbf08"
server_key_version: "encryptionkey@0"
I20260929 11:26:58.200976 28273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:58.220878 28273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:58.225342 28273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:58.225648 28273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:58.246390 28273 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:40691
I20260929 11:26:58.246403 28338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:40691 every 8 connection(s)
I20260929 11:26:58.247968 28273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:26:58.250967 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 28273
I20260929 11:26:58.251224 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:26:58.251817 24164 external_mini_cluster.cc:1428] Setting key 438e828d388c7f15df7adaf8c620459b
I20260929 11:26:58.252496 28339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 11:26:58 dist-test-slave-qk05 krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:58.270857 28339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05: Bootstrap starting.
I20260929 11:26:58.274746 28339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05: Neither blocks nor log segments found. Creating new log.
I20260929 11:26:58.276257 28339 log.cc:824] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05: Log is configured to *not* fsync() on all Append() calls
I20260929 11:26:58.280687 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:58.253978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59600 (local address 127.23.153.62:40691)
0929 11:26:58.255232 (+ 1254us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:58.255250 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:58.255310 (+ 60us) server_negotiation.cc:407] Connection header received
0929 11:26:58.257860 (+ 2550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:58.257912 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:58.258917 (+ 1005us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:58.259577 (+ 660us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:58.261149 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:58.264093 (+ 2944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:58.265849 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:58.266968 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:58.267250 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:58.271804 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:58.271819 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:58.271838 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:58.271886 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:58.275361 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:58.276497 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:58.276507 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:58.276514 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:58.276752 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:58.277266 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:58.277271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:58.277274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:58.278073 (+ 799us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:58.278319 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:58.279163 (+ 844us) server_negotiation.cc:299] Negotiation successful
0929 11:26:58.279583 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944,"thread_start_us":645,"threads_started":1}
I20260929 11:26:58.280766 28339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05: No bootstrap required, opened a new log
I20260929 11:26:58.289374 28339 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3559328b684408c81618658de962b05" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40691 } }
I20260929 11:26:58.289793 28339 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:26:58.289891 28339 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3559328b684408c81618658de962b05, State: Initialized, Role: FOLLOWER
I20260929 11:26:58.290526 28339 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3559328b684408c81618658de962b05" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40691 } }
I20260929 11:26:58.290762 28339 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:26:58.290896 28339 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:26:58.291093 28339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:26:58.293355 28339 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3559328b684408c81618658de962b05" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40691 } }
I20260929 11:26:58.293928 28339 leader_election.cc:302] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3559328b684408c81618658de962b05; no voters:
I20260929 11:26:58.294540 28339 leader_election.cc:288] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:26:58.294776 28344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:26:58.295729 28344 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05 [term 1 LEADER]: Becoming Leader. State: Replica: b3559328b684408c81618658de962b05, State: Running, Role: LEADER
I20260929 11:26:58.296283 28344 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3559328b684408c81618658de962b05" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40691 } }
I20260929 11:26:58.297027 28339 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:26:58.299394 28346 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3559328b684408c81618658de962b05. Latest consensus state: current_term: 1 leader_uuid: "b3559328b684408c81618658de962b05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3559328b684408c81618658de962b05" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40691 } } }
I20260929 11:26:58.299371 28345 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3559328b684408c81618658de962b05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3559328b684408c81618658de962b05" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40691 } } }
I20260929 11:26:58.299796 28346 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:58.299820 28345 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05 [sys.catalog]: This master's current role is: LEADER
I20260929 11:26:58.300591 28350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:26:58.305668 28350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:26:58.315775 28350 catalog_manager.cc:1347] Generated new cluster ID: 86ef5df2fac6491387a81b9091b090d4
I20260929 11:26:58.315908 28350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:26:58.327164 28350 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:26:58.329033 28350 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:26:58.338444 28350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05: Generated new TSK 0
I20260929 11:26:58.339548 28350 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
I20260929 11:26:58.398478 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40691
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:58.751205 28365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:58.751513 28365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:58.751617 28365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:58.759311 28365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:58.759451 28365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:58.759572 28365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:26:58.769906 28365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:58.771365 28365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:58.772676 28365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:58.786639 28371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:58.786736 28370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:58.786665 28373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:58.793375 28365 server_base.cc:1034] running on GCE node
I20260929 11:26:58.794296 28365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:58.795478 28365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:58.796744 28365 hybrid_clock.cc:648] HybridClock initialized: now 1790681218796676 us; error 62 us; skew 500 ppm
I20260929 11:26:58.797113 28365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:58 dist-test-slave-qk05 krb5kdc[28260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681218, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:58.801126 28365 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:26:58.802436 28365 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:58.802560 28365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:58.802867 28365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:58.806330 28365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "d8dec83a41b644f0896e85326b287fa0"
format_stamp: "Formatted at 2026-09-29 11:26:58 on dist-test-slave-qk05"
server_key: "a56d217e86a0780b9aa44115f36d5323"
server_key_iv: "59bb33bd345ff5950488fd5c6e7a0abe"
server_key_version: "encryptionkey@0"
I20260929 11:26:58.807298 28365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "d8dec83a41b644f0896e85326b287fa0"
format_stamp: "Formatted at 2026-09-29 11:26:58 on dist-test-slave-qk05"
server_key: "a56d217e86a0780b9aa44115f36d5323"
server_key_iv: "59bb33bd345ff5950488fd5c6e7a0abe"
server_key_version: "encryptionkey@0"
I20260929 11:26:58.813185 28365 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:26:58.817320 28379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:58.819103 28365 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:26:58.819388 28365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "d8dec83a41b644f0896e85326b287fa0"
format_stamp: "Formatted at 2026-09-29 11:26:58 on dist-test-slave-qk05"
server_key: "a56d217e86a0780b9aa44115f36d5323"
server_key_iv: "59bb33bd345ff5950488fd5c6e7a0abe"
server_key_version: "encryptionkey@0"
I20260929 11:26:58.819705 28365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:58.843569 28365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:58.847966 28365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:58.848199 28365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:58.849201 28365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:26:58.850613 28365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:58.850699 28365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:58.850781 28365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:58.850832 28365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:58.882011 28365 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:35667
I20260929 11:26:58.882062 28491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:35667 every 8 connection(s)
I20260929 11:26:58.882892 28365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:26:58.889678 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 28365
I20260929 11:26:58.889895 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:26:58.890480 24164 external_mini_cluster.cc:1428] Setting key 8f470b54ac8a5221b08e6b3fd9477909
Sep 29 11:26:58 dist-test-slave-qk05 krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681218, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:58.905694 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:58.885599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:38443 (local address 127.23.153.62:40691)
0929 11:26:58.886091 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:58.886095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:58.886999 (+ 904us) server_negotiation.cc:407] Connection header received
0929 11:26:58.889139 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:58.889144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:58.889258 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:58.889353 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:58.891512 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:58.892480 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:58.894264 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:58.894687 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:58.894806 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:58.899276 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:58.899283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:58.899286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:58.899324 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:58.901874 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:58.903047 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:58.903053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:58.903056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:58.903201 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:58.903737 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:58.903741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:58.903743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:58.904036 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:58.904149 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:58.905196 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0929 11:26:58.905384 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":278,"threads_started":1}
I20260929 11:26:58.907239 28495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:58.886025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:40691 (local address 127.23.153.1:38443)
0929 11:26:58.886806 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:58.886850 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:58.888788 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:58.889644 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:58.889654 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:58.890282 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:58.891301 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:58.891326 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:58.892650 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:58.892655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:58.894058 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:58.894074 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:58.894956 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:58.894964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:58.895225 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:58.896047 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:58.896077 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:58.898951 (+ 2874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:58.902090 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:58.902102 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:58.902122 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:58.902838 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:58.903312 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:58.903317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:58.903321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:58.903609 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:58.904167 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:58.904177 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:58.904684 (+ 507us) client_negotiation.cc:769] Sending connection context
0929 11:26:58.905016 (+ 332us) client_negotiation.cc:240] Negotiation successful
0929 11:26:58.905677 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":298,"threads_started":1}
WARNING: no policy specified for kudu/127.23.153.2@KRBTEST.COM; defaulting to no policy
I20260929 11:26:58.909018 28492 heartbeater.cc:347] Connected to a master server at 127.23.153.62:40691
I20260929 11:26:58.909380 28492 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:58.910072 28492 heartbeater.cc:511] Master 127.23.153.62:40691 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.2@KRBTEST.COM" created.
I20260929 11:26:58.913141 28304 ts_manager.cc:194] Registered new tserver with Master: d8dec83a41b644f0896e85326b287fa0 (127.23.153.1:35667)
I20260929 11:26:58.915277 28304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:38443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
I20260929 11:26:58.936371 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40691
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:59.280917 28498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:59.281248 28498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:59.281380 28498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:59.289083 28498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:59.289239 28498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:59.289377 28498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:26:59.299902 28498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:59.301378 28498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:59.302770 28498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:59.316577 28503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:59.316567 28504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:59.317032 28506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:59.317466 28498 server_base.cc:1034] running on GCE node
I20260929 11:26:59.318102 28498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:59.319151 28498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:59.320410 28498 hybrid_clock.cc:648] HybridClock initialized: now 1790681219320353 us; error 46 us; skew 500 ppm
I20260929 11:26:59.320818 28498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:59 dist-test-slave-qk05 krb5kdc[28260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681219, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:59.324572 28498 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:26:59.325876 28498 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:59.325984 28498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:59.326315 28498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:59.329521 28498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "da94e234b9a34842bbd82f9606ca0321"
format_stamp: "Formatted at 2026-09-29 11:26:59 on dist-test-slave-qk05"
server_key: "37970e8010c4659ac5e71b545c92cd16"
server_key_iv: "5527f96a91696f4912fe9f1cc5eaee75"
server_key_version: "encryptionkey@0"
I20260929 11:26:59.330439 28498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "da94e234b9a34842bbd82f9606ca0321"
format_stamp: "Formatted at 2026-09-29 11:26:59 on dist-test-slave-qk05"
server_key: "37970e8010c4659ac5e71b545c92cd16"
server_key_iv: "5527f96a91696f4912fe9f1cc5eaee75"
server_key_version: "encryptionkey@0"
I20260929 11:26:59.336351 28498 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 11:26:59.340750 28512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:59.342264 28498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:26:59.342453 28498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "da94e234b9a34842bbd82f9606ca0321"
format_stamp: "Formatted at 2026-09-29 11:26:59 on dist-test-slave-qk05"
server_key: "37970e8010c4659ac5e71b545c92cd16"
server_key_iv: "5527f96a91696f4912fe9f1cc5eaee75"
server_key_version: "encryptionkey@0"
I20260929 11:26:59.342684 28498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:59.358779 28498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:59.363476 28498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:59.363747 28498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:59.364898 28498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:26:59.366489 28498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:59.366597 28498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:59.366711 28498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:59.366780 28498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:59.398921 28498 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:44473
I20260929 11:26:59.398978 28624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:44473 every 8 connection(s)
I20260929 11:26:59.399832 28498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:26:59.407159 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 28498
I20260929 11:26:59.407343 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:26:59.407769 24164 external_mini_cluster.cc:1428] Setting key 1dbd24aa3aee4fb0efcd317e76b8e73c
Sep 29 11:26:59 dist-test-slave-qk05 krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681219, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:59.423892 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:59.402500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:53109 (local address 127.23.153.62:40691)
0929 11:26:59.402747 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:59.402751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:59.404146 (+ 1395us) server_negotiation.cc:407] Connection header received
0929 11:26:59.406234 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:59.406239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:59.406328 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:59.406425 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:59.408383 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:59.409315 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:59.411296 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:59.412002 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:59.412138 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:59.416700 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:59.416707 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:59.416710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:59.416751 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:59.419385 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:59.420501 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:59.420506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:59.420508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:59.420667 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:59.421323 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:59.421328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:59.421330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:59.421899 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:59.422155 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:59.423374 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0929 11:26:59.423635 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260929 11:26:59.425375 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:59.403140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:40691 (local address 127.23.153.2:53109)
0929 11:26:59.403927 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:59.403993 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:59.405896 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:59.406675 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:59.406684 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:59.407358 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:59.408147 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:59.408172 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:59.409504 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:59.409510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:59.411117 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:59.411134 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:59.412422 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:59.412429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:59.412791 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:59.413564 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:59.413597 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:59.416335 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:59.419584 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:59.419596 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:59.419619 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:59.420311 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:59.420783 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:59.420788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:59.420792 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:59.421089 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:59.422143 (+ 1054us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:59.422154 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:59.422706 (+ 552us) client_negotiation.cc:769] Sending connection context
0929 11:26:59.423125 (+ 419us) client_negotiation.cc:240] Negotiation successful
0929 11:26:59.423770 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":315,"threads_started":1}
WARNING: no policy specified for kudu/127.23.153.3@KRBTEST.COM; defaulting to no policy
I20260929 11:26:59.427222 28625 heartbeater.cc:347] Connected to a master server at 127.23.153.62:40691
I20260929 11:26:59.427577 28625 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:59.428345 28625 heartbeater.cc:511] Master 127.23.153.62:40691 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.3@KRBTEST.COM" created.
I20260929 11:26:59.430054 28304 ts_manager.cc:194] Registered new tserver with Master: da94e234b9a34842bbd82f9606ca0321 (127.23.153.2:44473)
I20260929 11:26:59.431236 28304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:53109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
I20260929 11:26:59.452198 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40691
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:36941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:26:59.798516 28630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:26:59.798858 28630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:26:59.798995 28630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:26:59.806996 28630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:26:59.807178 28630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:26:59.807339 28630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:26:59.818089 28630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:26:59.819536 28630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:26:59.820914 28630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:26:59.835532 28635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:59.836421 28636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:26:59.837137 28638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:26:59.837778 28630 server_base.cc:1034] running on GCE node
I20260929 11:26:59.838467 28630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:26:59.839601 28630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:26:59.840857 28630 hybrid_clock.cc:648] HybridClock initialized: now 1790681219840779 us; error 57 us; skew 500 ppm
I20260929 11:26:59.841262 28630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:26:59 dist-test-slave-qk05 krb5kdc[28260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681219, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:26:59.844823 28630 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:26:59.846187 28630 fs_manager.cc:359] Metadata directory not provided
I20260929 11:26:59.846309 28630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:26:59.846665 28630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:26:59.849949 28630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "eb54e913947540c59150817114a97b3b"
format_stamp: "Formatted at 2026-09-29 11:26:59 on dist-test-slave-qk05"
server_key: "4a3ce87703793c0045f83506580654e1"
server_key_iv: "c70628f7e08e8ed35ecef1d646daf266"
server_key_version: "encryptionkey@0"
I20260929 11:26:59.850869 28630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "eb54e913947540c59150817114a97b3b"
format_stamp: "Formatted at 2026-09-29 11:26:59 on dist-test-slave-qk05"
server_key: "4a3ce87703793c0045f83506580654e1"
server_key_iv: "c70628f7e08e8ed35ecef1d646daf266"
server_key_version: "encryptionkey@0"
I20260929 11:26:59.856739 28630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 11:26:59.861172 28644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:59.862900 28630 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 11:26:59.863176 28630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "eb54e913947540c59150817114a97b3b"
format_stamp: "Formatted at 2026-09-29 11:26:59 on dist-test-slave-qk05"
server_key: "4a3ce87703793c0045f83506580654e1"
server_key_iv: "c70628f7e08e8ed35ecef1d646daf266"
server_key_version: "encryptionkey@0"
I20260929 11:26:59.863421 28630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:26:59.885425 28630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:26:59.890339 28630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:26:59.890630 28630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:26:59.891738 28630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:26:59.893220 28630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:26:59.893433 28630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:59.893687 28630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:26:59.893780 28630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:26:59.919299 28492 heartbeater.cc:503] Master 127.23.153.62:40691 was elected leader, sending a full tablet report...
I20260929 11:26:59.927357 28630 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:36725
I20260929 11:26:59.927412 28756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:36725 every 8 connection(s)
I20260929 11:26:59.928262 28630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:26:59.932142 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 28630
I20260929 11:26:59.932358 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:26:59.932870 24164 external_mini_cluster.cc:1428] Setting key 6016c25d2953162a6fd21f2c722c7ecb
Sep 29 11:26:59 dist-test-slave-qk05 krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681219, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:26:59.950673 28759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:59.931091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:47883 (local address 127.23.153.62:40691)
0929 11:26:59.931663 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:59.931669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:59.932785 (+ 1116us) server_negotiation.cc:407] Connection header received
0929 11:26:59.934967 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:59.934973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:59.935082 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:59.935173 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:59.936965 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:59.937733 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:59.939466 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:59.939923 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:59.940049 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:59.944488 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:59.944494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:59.944497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:59.944536 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:59.947461 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:59.948416 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:59.948422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:59.948425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:59.948582 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:59.949066 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:59.949069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:59.949072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:59.949355 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:59.949485 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:59.950181 (+ 696us) server_negotiation.cc:299] Negotiation successful
0929 11:26:59.950381 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":221,"threads_started":1}
I20260929 11:26:59.951717 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:59.931771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:40691 (local address 127.23.153.3:47883)
0929 11:26:59.932620 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0929 11:26:59.932667 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 11:26:59.934636 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:26:59.935449 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:26:59.935459 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:26:59.936120 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:26:59.936757 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:59.936778 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:59.937896 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:59.937900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:59.939294 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:26:59.939310 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:26:59.940177 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:26:59.940184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:26:59.940413 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:59.941079 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:26:59.941102 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:26:59.944161 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:26:59.947630 (+ 3469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:59.947642 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:59.947659 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:59.948246 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:59.948706 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:26:59.948711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:26:59.948715 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:26:59.948945 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:26:59.949458 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:26:59.949464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:26:59.949823 (+ 359us) client_negotiation.cc:769] Sending connection context
0929 11:26:59.950061 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 11:26:59.950499 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":326,"threads_started":1}
I20260929 11:26:59.953562 28757 heartbeater.cc:347] Connected to a master server at 127.23.153.62:40691
I20260929 11:26:59.953892 28757 heartbeater.cc:464] Registering TS with master...
I20260929 11:26:59.954569 28757 heartbeater.cc:511] Master 127.23.153.62:40691 requested a full tablet report, sending...
I20260929 11:26:59.956175 28304 ts_manager.cc:194] Registered new tserver with Master: eb54e913947540c59150817114a97b3b (127.23.153.3:36725)
I20260929 11:26:59.956986 28304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:47883
I20260929 11:26:59.961465 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:26:59.979905 28759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:26:59.966253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59612 (local address 127.23.153.62:40691)
0929 11:26:59.966635 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0929 11:26:59.966642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:26:59.967038 (+ 396us) server_negotiation.cc:407] Connection header received
0929 11:26:59.967261 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:26:59.967265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:26:59.967334 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:26:59.967418 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:26:59.968905 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:59.969614 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:59.970988 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:26:59.971431 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:26:59.971856 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:26:59.974575 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:26:59.974581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:26:59.974585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:26:59.974628 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:26:59.977187 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:59.977922 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:59.977927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:59.977930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:59.978065 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:26:59.978416 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:26:59.978420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:26:59.978422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:26:59.978751 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:26:59.978990 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 11:26:59.979352 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 11:26:59.979597 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260929 11:26:59.984907 28304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:26:59.994225 28304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:27:00.034181 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:00.009612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:35667 (local address 127.0.0.1:49348)
0929 11:27:00.025418 (+ 15806us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:00.025468 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:00.025772 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:00.027275 (+ 1503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:00.027282 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:00.027314 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:00.027749 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:00.027766 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.029529 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.029534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:00.031642 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:00.031655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.033071 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.033077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:00.033303 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:00.033356 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 11:27:00.033505 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 11:27:00.033727 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15461,"mutex_wait_us":2,"thread_start_us":16506,"threads_started":1}
I20260929 11:27:00.035050 28770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:00.009714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.23.153.1:35667)
0929 11:27:00.025094 (+ 15380us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:00.025104 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:00.026166 (+ 1062us) server_negotiation.cc:407] Connection header received
0929 11:27:00.026289 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:00.026298 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:00.026699 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:00.026910 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:00.027989 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.029337 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.031821 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.032819 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.032965 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:00.034432 (+ 1467us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:00.034543 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 11:27:00.034743 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15115,"thread_start_us":15409,"threads_started":1}
I20260929 11:27:00.036548 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:00.010787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:36725 (local address 127.0.0.1:47748)
0929 11:27:00.026422 (+ 15635us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:00.026445 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:00.026579 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:00.027556 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:00.027560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:00.027583 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:00.028009 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:00.028020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.030813 (+ 2793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.030818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:00.033778 (+ 2960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:00.033787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.035786 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.035792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:00.035977 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:00.036015 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 11:27:00.036121 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 11:27:00.036248 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15507,"mutex_wait_us":146,"thread_start_us":13387,"threads_started":1}
I20260929 11:27:00.036935 28771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:00.009661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:44473 (local address 127.0.0.1:49372)
0929 11:27:00.025806 (+ 16145us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:00.025827 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:00.026000 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:00.028591 (+ 2591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:00.028596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:00.028624 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:00.028931 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:00.028943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.030729 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.030735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:00.033766 (+ 3031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:00.033785 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.036060 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.036065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:00.036248 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:00.036279 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 11:27:00.036378 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 11:27:00.036541 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16008,"mutex_wait_us":206,"thread_start_us":15828,"threads_started":1}
I20260929 11:27:00.037128 28772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:00.009841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49372 (local address 127.23.153.2:44473)
0929 11:27:00.027935 (+ 18094us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:00.027944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:00.027984 (+ 40us) server_negotiation.cc:407] Connection header received
0929 11:27:00.028059 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:00.028063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:00.028307 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:00.028620 (+ 313us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:00.029112 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.030543 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.034023 (+ 3480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.035566 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.035708 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:00.035907 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:00.036660 (+ 753us) server_negotiation.cc:299] Negotiation successful
0929 11:27:00.036842 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17946,"thread_start_us":17849,"threads_started":1}
I20260929 11:27:00.037772 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:00.011541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47748 (local address 127.23.153.3:36725)
0929 11:27:00.018547 (+ 7006us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:00.018557 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:00.026562 (+ 8005us) server_negotiation.cc:407] Connection header received
0929 11:27:00.026791 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:00.026800 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:00.027089 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:00.027267 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:00.028249 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.030669 (+ 2420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.034011 (+ 3342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.035566 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.035709 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:00.035945 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:00.037329 (+ 1384us) server_negotiation.cc:299] Negotiation successful
0929 11:27:00.037485 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6809,"thread_start_us":6635,"threads_started":1}
I20260929 11:27:00.039561 28427 tablet_service.cc:1467] Processing CreateTablet for tablet f7da4c2c63ea4d09ad2ef6066916bc87 (DEFAULT_TABLE table=test-table [id=d20a85ffbcf24c1395c57871e96e5ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:00.041201 28427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7da4c2c63ea4d09ad2ef6066916bc87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:00.040874 28692 tablet_service.cc:1467] Processing CreateTablet for tablet f7da4c2c63ea4d09ad2ef6066916bc87 (DEFAULT_TABLE table=test-table [id=d20a85ffbcf24c1395c57871e96e5ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:00.040959 28560 tablet_service.cc:1467] Processing CreateTablet for tablet f7da4c2c63ea4d09ad2ef6066916bc87 (DEFAULT_TABLE table=test-table [id=d20a85ffbcf24c1395c57871e96e5ec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:00.042127 28692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7da4c2c63ea4d09ad2ef6066916bc87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:00.042287 28560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7da4c2c63ea4d09ad2ef6066916bc87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:00.054224 28776 tablet_bootstrap.cc:492] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b: Bootstrap starting.
I20260929 11:27:00.058241 28776 tablet_bootstrap.cc:654] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:00.059149 28778 tablet_bootstrap.cc:492] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0: Bootstrap starting.
I20260929 11:27:00.059983 28776 log.cc:824] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:00.061722 28777 tablet_bootstrap.cc:492] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321: Bootstrap starting.
I20260929 11:27:00.063189 28778 tablet_bootstrap.cc:654] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:00.064724 28776 tablet_bootstrap.cc:492] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b: No bootstrap required, opened a new log
I20260929 11:27:00.064711 28778 log.cc:824] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:00.065073 28776 ts_tablet_manager.cc:1397] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 11:27:00.067735 28777 tablet_bootstrap.cc:654] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:00.069422 28777 log.cc:824] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:00.069747 28778 tablet_bootstrap.cc:492] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0: No bootstrap required, opened a new log
I20260929 11:27:00.070358 28778 ts_tablet_manager.cc:1397] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260929 11:27:00.073768 28777 tablet_bootstrap.cc:492] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321: No bootstrap required, opened a new log
I20260929 11:27:00.074149 28777 ts_tablet_manager.cc:1397] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260929 11:27:00.074281 28776 raft_consensus.cc:348] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dec83a41b644f0896e85326b287fa0" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35667 } } peers { permanent_uuid: "da94e234b9a34842bbd82f9606ca0321" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44473 } } peers { permanent_uuid: "eb54e913947540c59150817114a97b3b" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36725 } }
I20260929 11:27:00.074676 28776 raft_consensus.cc:374] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:00.074770 28776 raft_consensus.cc:729] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb54e913947540c59150817114a97b3b, State: Initialized, Role: FOLLOWER
I20260929 11:27:00.075412 28776 consensus_queue.cc:260] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dec83a41b644f0896e85326b287fa0" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35667 } } peers { permanent_uuid: "da94e234b9a34842bbd82f9606ca0321" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44473 } } peers { permanent_uuid: "eb54e913947540c59150817114a97b3b" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36725 } }
I20260929 11:27:00.076601 28757 heartbeater.cc:503] Master 127.23.153.62:40691 was elected leader, sending a full tablet report...
I20260929 11:27:00.079165 28776 ts_tablet_manager.cc:1428] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260929 11:27:00.081292 28778 raft_consensus.cc:348] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dec83a41b644f0896e85326b287fa0" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35667 } } peers { permanent_uuid: "da94e234b9a34842bbd82f9606ca0321" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44473 } } peers { permanent_uuid: "eb54e913947540c59150817114a97b3b" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36725 } }
I20260929 11:27:00.081820 28778 raft_consensus.cc:374] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:00.081928 28778 raft_consensus.cc:729] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8dec83a41b644f0896e85326b287fa0, State: Initialized, Role: FOLLOWER
I20260929 11:27:00.082640 28778 consensus_queue.cc:260] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dec83a41b644f0896e85326b287fa0" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35667 } } peers { permanent_uuid: "da94e234b9a34842bbd82f9606ca0321" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44473 } } peers { permanent_uuid: "eb54e913947540c59150817114a97b3b" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36725 } }
I20260929 11:27:00.084993 28778 ts_tablet_manager.cc:1428] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260929 11:27:00.084928 28777 raft_consensus.cc:348] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dec83a41b644f0896e85326b287fa0" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35667 } } peers { permanent_uuid: "da94e234b9a34842bbd82f9606ca0321" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44473 } } peers { permanent_uuid: "eb54e913947540c59150817114a97b3b" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36725 } }
I20260929 11:27:00.085444 28777 raft_consensus.cc:374] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:00.085546 28777 raft_consensus.cc:729] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da94e234b9a34842bbd82f9606ca0321, State: Initialized, Role: FOLLOWER
I20260929 11:27:00.086093 28777 consensus_queue.cc:260] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dec83a41b644f0896e85326b287fa0" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35667 } } peers { permanent_uuid: "da94e234b9a34842bbd82f9606ca0321" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44473 } } peers { permanent_uuid: "eb54e913947540c59150817114a97b3b" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36725 } }
I20260929 11:27:00.087183 28625 heartbeater.cc:503] Master 127.23.153.62:40691 was elected leader, sending a full tablet report...
I20260929 11:27:00.087337 28784 raft_consensus.cc:482] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:27:00.087656 28784 raft_consensus.cc:504] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dec83a41b644f0896e85326b287fa0" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35667 } } peers { permanent_uuid: "da94e234b9a34842bbd82f9606ca0321" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44473 } } peers { permanent_uuid: "eb54e913947540c59150817114a97b3b" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36725 } }
I20260929 11:27:00.088020 28777 ts_tablet_manager.cc:1428] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260929 11:27:00.090164 28784 leader_election.cc:288] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8dec83a41b644f0896e85326b287fa0 (127.23.153.1:35667), eb54e913947540c59150817114a97b3b (127.23.153.3:36725)
I20260929 11:27:00.096817 28770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:00.089964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:57385 (local address 127.23.153.1:35667)
0929 11:27:00.090152 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:00.090158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:00.090895 (+ 737us) server_negotiation.cc:407] Connection header received
0929 11:27:00.091096 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:00.091102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:00.091188 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:00.091264 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:00.091782 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.092689 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.095089 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.095878 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.096036 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:00.096115 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:00.096411 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 11:27:00.096576 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:27:00.096870 28785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:00.089801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:35667 (local address 127.23.153.2:57385)
0929 11:27:00.090793 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:00.090808 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:00.090944 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:00.091297 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:00.091301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:00.091328 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:00.091646 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:00.091656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.092847 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.092853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:00.094889 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:00.094905 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.096115 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.096122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:00.096257 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:00.096282 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 11:27:00.096371 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 11:27:00.096514 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"thread_start_us":699,"threads_started":1}
I20260929 11:27:00.098030 28447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7da4c2c63ea4d09ad2ef6066916bc87" candidate_uuid: "da94e234b9a34842bbd82f9606ca0321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8dec83a41b644f0896e85326b287fa0" is_pre_election: true
I20260929 11:27:00.098672 28447 raft_consensus.cc:2454] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da94e234b9a34842bbd82f9606ca0321 in term 0.
I20260929 11:27:00.099591 28516 leader_election.cc:302] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8dec83a41b644f0896e85326b287fa0, da94e234b9a34842bbd82f9606ca0321; no voters:
I20260929 11:27:00.100104 28784 raft_consensus.cc:2781] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:27:00.100301 28784 raft_consensus.cc:482] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:27:00.100409 28784 raft_consensus.cc:3037] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:00.103190 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:00.090273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:59875 (local address 127.23.153.3:36725)
0929 11:27:00.090432 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:00.090444 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:00.097515 (+ 7071us) server_negotiation.cc:407] Connection header received
0929 11:27:00.097716 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:00.097721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:00.097801 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:00.097899 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:00.098188 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.099290 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.101251 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.102053 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.102243 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:00.102319 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:00.102760 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 11:27:00.102893 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 11:27:00.103211 28785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:00.089888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:36725 (local address 127.23.153.2:59875)
0929 11:27:00.097390 (+ 7502us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:00.097409 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:00.097551 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:00.097906 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:00.097910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:00.097933 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:00.098065 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:00.098072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.099457 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.099463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:00.101073 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:00.101089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.102420 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.102425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:00.102595 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:00.102622 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 11:27:00.102713 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 11:27:00.102871 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7399,"threads_started":1}
I20260929 11:27:00.104589 28784 raft_consensus.cc:504] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dec83a41b644f0896e85326b287fa0" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35667 } } peers { permanent_uuid: "da94e234b9a34842bbd82f9606ca0321" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44473 } } peers { permanent_uuid: "eb54e913947540c59150817114a97b3b" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36725 } }
I20260929 11:27:00.105418 28784 leader_election.cc:288] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [CANDIDATE]: Term 1 election: Requested vote from peers d8dec83a41b644f0896e85326b287fa0 (127.23.153.1:35667), eb54e913947540c59150817114a97b3b (127.23.153.3:36725)
I20260929 11:27:00.105882 28447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7da4c2c63ea4d09ad2ef6066916bc87" candidate_uuid: "da94e234b9a34842bbd82f9606ca0321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8dec83a41b644f0896e85326b287fa0"
I20260929 11:27:00.106101 28447 raft_consensus.cc:3037] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:00.105919 28712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7da4c2c63ea4d09ad2ef6066916bc87" candidate_uuid: "da94e234b9a34842bbd82f9606ca0321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb54e913947540c59150817114a97b3b" is_pre_election: true
I20260929 11:27:00.106107 28711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7da4c2c63ea4d09ad2ef6066916bc87" candidate_uuid: "da94e234b9a34842bbd82f9606ca0321" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb54e913947540c59150817114a97b3b"
I20260929 11:27:00.106676 28712 raft_consensus.cc:2454] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da94e234b9a34842bbd82f9606ca0321 in term 0.
I20260929 11:27:00.108807 28447 raft_consensus.cc:2454] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da94e234b9a34842bbd82f9606ca0321 in term 1.
I20260929 11:27:00.109232 28516 leader_election.cc:302] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d8dec83a41b644f0896e85326b287fa0, da94e234b9a34842bbd82f9606ca0321; no voters: eb54e913947540c59150817114a97b3b
I20260929 11:27:00.109592 28784 raft_consensus.cc:2781] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:00.110018 28784 raft_consensus.cc:686] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [term 1 LEADER]: Becoming Leader. State: Replica: da94e234b9a34842bbd82f9606ca0321, State: Running, Role: LEADER
I20260929 11:27:00.110517 28784 consensus_queue.cc:237] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dec83a41b644f0896e85326b287fa0" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35667 } } peers { permanent_uuid: "da94e234b9a34842bbd82f9606ca0321" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44473 } } peers { permanent_uuid: "eb54e913947540c59150817114a97b3b" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36725 } }
I20260929 11:27:00.115396 28304 catalog_manager.cc:5495] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 reported cstate change: term changed from 0 to 1, leader changed from <none> to da94e234b9a34842bbd82f9606ca0321 (127.23.153.2). New cstate: current_term: 1 leader_uuid: "da94e234b9a34842bbd82f9606ca0321" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dec83a41b644f0896e85326b287fa0" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da94e234b9a34842bbd82f9606ca0321" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb54e913947540c59150817114a97b3b" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36725 } health_report { overall_health: UNKNOWN } } }
W20260929 11:27:00.135314 28493 tablet.cc:2369] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:27:00.142529 28772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:00.137070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49382 (local address 127.23.153.2:44473)
0929 11:27:00.137242 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:00.137247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:00.137266 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:27:00.137394 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:00.137398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:00.137571 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:00.137733 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:27:00.138058 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.138753 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.140213 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.140782 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.140888 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:00.141385 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:27:00.141927 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:27:00.142044 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:00.142185 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 11:27:00.142264 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:27:00.151320 28712 raft_consensus.cc:3037] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:00.151702 28447 raft_consensus.cc:1261] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0 [term 1 FOLLOWER]: Refusing update from remote peer da94e234b9a34842bbd82f9606ca0321: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 11:27:00.151750 28626 tablet.cc:2369] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:27:00.152928 28787 consensus_queue.cc:1035] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8dec83a41b644f0896e85326b287fa0" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:00.154688 28712 raft_consensus.cc:1261] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b [term 1 FOLLOWER]: Refusing update from remote peer da94e234b9a34842bbd82f9606ca0321: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:27:00.155606 28787 consensus_queue.cc:1035] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb54e913947540c59150817114a97b3b" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:00.166006 28790 mvcc.cc:205] Tried to move back new op lower bound from 7334630277729202176 to 7334630277577252864. Current Snapshot: MvccSnapshot[applied={T|T < 7334630277729202176}]
I20260929 11:27:00.173051 28791 mvcc.cc:205] Tried to move back new op lower bound from 7334630277729202176 to 7334630277577252864. Current Snapshot: MvccSnapshot[applied={T|T < 7334630277729202176}]
I20260929 11:27:00.182189 28304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:27:00.182437 28304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:27:00.186326 28304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3559328b684408c81618658de962b05: Sending DeleteTablet for 3 replicas of tablet f7da4c2c63ea4d09ad2ef6066916bc87
I20260929 11:27:00.187804 28692 tablet_service.cc:1514] Processing DeleteTablet for tablet f7da4c2c63ea4d09ad2ef6066916bc87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:00 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:47748
I20260929 11:27:00.187767 28560 tablet_service.cc:1514] Processing DeleteTablet for tablet f7da4c2c63ea4d09ad2ef6066916bc87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:00 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:49372
I20260929 11:27:00.188006 28427 tablet_service.cc:1514] Processing DeleteTablet for tablet f7da4c2c63ea4d09ad2ef6066916bc87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:00 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:49348
I20260929 11:27:00.188874 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 28365
I20260929 11:27:00.197258 28800 tablet_replica.cc:330] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b: stopping tablet replica
I20260929 11:27:00.198266 28799 tablet_replica.cc:330] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321: stopping tablet replica
I20260929 11:27:00.198750 28799 raft_consensus.cc:2229] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:27:00.199551 28799 raft_consensus.cc:2258] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:00.200215 28800 raft_consensus.cc:2229] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:27:00.200830 28800 raft_consensus.cc:2258] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:00.205084 28801 tablet_replica.cc:330] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0: stopping tablet replica
I20260929 11:27:00.205911 28800 ts_tablet_manager.cc:1905] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:00.205904 28801 raft_consensus.cc:2229] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:27:00.206463 28801 raft_consensus.cc:2258] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:00.209863 28801 ts_tablet_manager.cc:1905] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 11:27:00 dist-test-slave-qk05 krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:27:00.213982 28800 ts_tablet_manager.cc:1918] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:27:00.214188 28800 log.cc:1196] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/f7da4c2c63ea4d09ad2ef6066916bc87
I20260929 11:27:00.214730 28800 ts_tablet_manager.cc:1939] T f7da4c2c63ea4d09ad2ef6066916bc87 P eb54e913947540c59150817114a97b3b: Deleting consensus metadata
I20260929 11:27:00.215204 28799 ts_tablet_manager.cc:1905] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:00.218171 28801 ts_tablet_manager.cc:1918] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:27:00.218367 28801 log.cc:1196] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/f7da4c2c63ea4d09ad2ef6066916bc87
I20260929 11:27:00.218858 28801 ts_tablet_manager.cc:1939] T f7da4c2c63ea4d09ad2ef6066916bc87 P d8dec83a41b644f0896e85326b287fa0: Deleting consensus metadata
I20260929 11:27:00.221053 28291 catalog_manager.cc:4841] TS d8dec83a41b644f0896e85326b287fa0 (127.23.153.1:35667): tablet f7da4c2c63ea4d09ad2ef6066916bc87 (table test-table [id=d20a85ffbcf24c1395c57871e96e5ec1]) successfully deleted
I20260929 11:27:00.222035 28770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:00.190338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.23.153.1:35667)
0929 11:27:00.190545 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:00.190552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:00.197338 (+ 6786us) server_negotiation.cc:407] Connection header received
0929 11:27:00.198144 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:00.198152 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:00.198245 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:00.198323 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:00.200682 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.201881 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.204718 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.205701 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.205835 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:00.212859 (+ 7024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:00.212881 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:00.212902 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:00.212963 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:00.217741 (+ 4778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:00.219166 (+ 1425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:00.219179 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:00.219183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:00.219382 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:00.219901 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:00.219907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:00.219910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:00.220391 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:00.220721 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:00.221389 (+ 668us) server_negotiation.cc:299] Negotiation successful
0929 11:27:00.221652 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:27:00.223049 28799 ts_tablet_manager.cc:1918] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:27:00.223156 28288 catalog_manager.cc:4841] TS eb54e913947540c59150817114a97b3b (127.23.153.3:36725): tablet f7da4c2c63ea4d09ad2ef6066916bc87 (table test-table [id=d20a85ffbcf24c1395c57871e96e5ec1]) successfully deleted
I20260929 11:27:00.223251 28799 log.cc:1196] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/f7da4c2c63ea4d09ad2ef6066916bc87
I20260929 11:27:00.224097 28799 ts_tablet_manager.cc:1939] T f7da4c2c63ea4d09ad2ef6066916bc87 P da94e234b9a34842bbd82f9606ca0321: Deleting consensus metadata
I20260929 11:27:00.224370 28487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:00.226099 28289 catalog_manager.cc:4841] TS da94e234b9a34842bbd82f9606ca0321 (127.23.153.2:44473): tablet f7da4c2c63ea4d09ad2ef6066916bc87 (table test-table [id=d20a85ffbcf24c1395c57871e96e5ec1]) successfully deleted
I20260929 11:27:00.329716 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 28365
I20260929 11:27:00.345504 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 28498
Sep 29 11:27:00 dist-test-slave-qk05 krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.2@KRBTEST.COM
I20260929 11:27:00.366307 28772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:00.346285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49384 (local address 127.23.153.2:44473)
0929 11:27:00.346695 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:00.346703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:00.346792 (+ 89us) server_negotiation.cc:407] Connection header received
0929 11:27:00.347119 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:00.347124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:00.347229 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:00.347368 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:00.349311 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.350540 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.352651 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.353392 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.353567 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:00.358939 (+ 5372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:00.358957 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:00.358974 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:00.359199 (+ 225us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:00.362374 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:00.363217 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:00.363225 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:00.363227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:00.363405 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:00.364235 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:00.364241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:00.364243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:00.364809 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:00.365268 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:00.365553 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 11:27:00.365859 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:27:00.367211 28620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:00.486881 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 28498
I20260929 11:27:00.502797 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 28630
Sep 29 11:27:00 dist-test-slave-qk05 krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.3@KRBTEST.COM
I20260929 11:27:00.519583 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:00.503642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47750 (local address 127.23.153.3:36725)
0929 11:27:00.503819 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:00.503823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:00.503844 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:27:00.503966 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:00.503970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:00.504034 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:00.504123 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:00.505443 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.506256 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.507875 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:00.508298 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:00.508427 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:00.513381 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:00.513400 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:00.513418 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:00.513474 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:00.516432 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:00.517255 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:00.517263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:00.517266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:00.517424 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:00.517827 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:00.517830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:00.517832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:00.518233 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:00.518471 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:00.518762 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 11:27:00.518954 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 11:27:00.520203 28752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:00.613396 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 28630
I20260929 11:27:00.627187 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 28273
I20260929 11:27:00.628568 28334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:00.745630 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 28273
2026-09-29T11:27:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4189 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T11:27:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:27:00Z Disabled control of system clock
I20260929 11:27:00.816498 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:42533
--webserver_interface=127.23.153.62
--webserver_port=0
--builtin_ntp_servers=127.23.153.20:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:42533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 11:27:01.177125 28810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:01.177444 28810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:01.177528 28810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:01.185185 28810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:27:01.185329 28810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:01.185366 28810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:01.185427 28810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:27:01.185470 28810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:27:01.194944 28810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:42533
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:42533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:01.196264 28810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:01.198102 28810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:01.210482 28815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:01.210421 28816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:01.210451 28818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:01.211301 28810 server_base.cc:1034] running on GCE node
I20260929 11:27:01.212210 28810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:01.213845 28810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:01.215176 28810 hybrid_clock.cc:648] HybridClock initialized: now 1790681221215115 us; error 58 us; skew 500 ppm
I20260929 11:27:01.215592 28810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:27:01.223389 28810 webserver.cc:460] Webserver started at http://127.23.153.62:36675/ using document root <none> and password file <none>
I20260929 11:27:01.224215 28810 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:01.224324 28810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:01.224673 28810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:01.228041 28810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "4b139c75c38747f5b8f4997da3254bbf"
format_stamp: "Formatted at 2026-09-29 11:27:01 on dist-test-slave-qk05"
server_key: "2a0f54c890789dc9923c4e7ff27256b1"
server_key_iv: "3f3094a3b18616c3994ea59389c272c2"
server_key_version: "encryptionkey@0"
I20260929 11:27:01.228925 28810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "4b139c75c38747f5b8f4997da3254bbf"
format_stamp: "Formatted at 2026-09-29 11:27:01 on dist-test-slave-qk05"
server_key: "2a0f54c890789dc9923c4e7ff27256b1"
server_key_iv: "3f3094a3b18616c3994ea59389c272c2"
server_key_version: "encryptionkey@0"
I20260929 11:27:01.234670 28810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 11:27:01.238626 28824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:01.240423 28810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 11:27:01.240641 28810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "4b139c75c38747f5b8f4997da3254bbf"
format_stamp: "Formatted at 2026-09-29 11:27:01 on dist-test-slave-qk05"
server_key: "2a0f54c890789dc9923c4e7ff27256b1"
server_key_iv: "3f3094a3b18616c3994ea59389c272c2"
server_key_version: "encryptionkey@0"
I20260929 11:27:01.240887 28810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:01.248099 28810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:01.249346 28810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:01.249794 28810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:01.270262 28810 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:42533
I20260929 11:27:01.270288 28875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:42533 every 8 connection(s)
I20260929 11:27:01.271848 28810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:27:01.276015 28876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:01.278822 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 28810
I20260929 11:27:01.279085 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:27:01.279922 24164 external_mini_cluster.cc:1428] Setting key 00257ee2ba52b7e3b8166455d8587c9b
I20260929 11:27:01.290091 28879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:01.282281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33108 (local address 127.23.153.62:42533)
0929 11:27:01.283483 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:01.283541 (+ 58us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:01.283637 (+ 96us) server_negotiation.cc:407] Connection header received
0929 11:27:01.285915 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:01.285948 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:01.286578 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:01.287132 (+ 554us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:01.288280 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:01.288287 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:01.288302 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:01.288324 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:01.288396 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:01.288415 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:01.288651 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:01.288796 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 11:27:01.289094 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":382,"threads_started":1}
I20260929 11:27:01.290216 28876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf: Bootstrap starting.
I20260929 11:27:01.293670 28876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:01.294844 28876 log.cc:824] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:01.297844 28876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf: No bootstrap required, opened a new log
I20260929 11:27:01.306180 28876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b139c75c38747f5b8f4997da3254bbf" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42533 } }
I20260929 11:27:01.306802 28876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:01.306918 28876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b139c75c38747f5b8f4997da3254bbf, State: Initialized, Role: FOLLOWER
I20260929 11:27:01.307766 28876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b139c75c38747f5b8f4997da3254bbf" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42533 } }
I20260929 11:27:01.308050 28876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:27:01.308207 28876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:27:01.308466 28876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:01.310859 28876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b139c75c38747f5b8f4997da3254bbf" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42533 } }
I20260929 11:27:01.311533 28876 leader_election.cc:302] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b139c75c38747f5b8f4997da3254bbf; no voters:
I20260929 11:27:01.312208 28876 leader_election.cc:288] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:27:01.312371 28881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:01.313056 28881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf [term 1 LEADER]: Becoming Leader. State: Replica: 4b139c75c38747f5b8f4997da3254bbf, State: Running, Role: LEADER
I20260929 11:27:01.313732 28881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b139c75c38747f5b8f4997da3254bbf" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42533 } }
I20260929 11:27:01.314394 28876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:27:01.316870 28883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b139c75c38747f5b8f4997da3254bbf. Latest consensus state: current_term: 1 leader_uuid: "4b139c75c38747f5b8f4997da3254bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b139c75c38747f5b8f4997da3254bbf" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42533 } } }
I20260929 11:27:01.317334 28883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:01.316915 28882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b139c75c38747f5b8f4997da3254bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b139c75c38747f5b8f4997da3254bbf" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42533 } } }
I20260929 11:27:01.317457 28882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:01.319489 28888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:27:01.324765 28888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:27:01.334954 28888 catalog_manager.cc:1347] Generated new cluster ID: 2a69663fe917454dac53d32df080c8cf
I20260929 11:27:01.335196 28888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:27:01.353577 28888 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:27:01.354058 28888 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:27:01.361789 28888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf: Generated new TSK 0
I20260929 11:27:01.363081 28888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:27:01.377692 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42533
--builtin_ntp_servers=127.23.153.20:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 11:27:01.729207 28900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:01.729528 28900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:01.729640 28900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:01.737231 28900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:01.737356 28900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:01.737498 28900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:27:01.747058 28900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:01.748412 28900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:01.750272 28900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:01.764279 28906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:01.764245 28908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:01.764212 28905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:01.765306 28900 server_base.cc:1034] running on GCE node
I20260929 11:27:01.765933 28900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:01.766939 28900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:01.768229 28900 hybrid_clock.cc:648] HybridClock initialized: now 1790681221768163 us; error 47 us; skew 500 ppm
I20260929 11:27:01.768611 28900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:27:01.771687 28900 webserver.cc:460] Webserver started at http://127.23.153.1:46123/ using document root <none> and password file <none>
I20260929 11:27:01.772485 28900 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:01.772596 28900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:01.772950 28900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:01.776352 28900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "0ad1aba11e9a4a41a2f99fc71d5ab0a7"
format_stamp: "Formatted at 2026-09-29 11:27:01 on dist-test-slave-qk05"
server_key: "0df74fe2824067186dbdc155ea07aa30"
server_key_iv: "cc1c19f7e2470377358c81c4a88e6482"
server_key_version: "encryptionkey@0"
I20260929 11:27:01.777264 28900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "0ad1aba11e9a4a41a2f99fc71d5ab0a7"
format_stamp: "Formatted at 2026-09-29 11:27:01 on dist-test-slave-qk05"
server_key: "0df74fe2824067186dbdc155ea07aa30"
server_key_iv: "cc1c19f7e2470377358c81c4a88e6482"
server_key_version: "encryptionkey@0"
I20260929 11:27:01.783186 28900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 11:27:01.787259 28914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:01.789008 28900 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:27:01.789211 28900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "0ad1aba11e9a4a41a2f99fc71d5ab0a7"
format_stamp: "Formatted at 2026-09-29 11:27:01 on dist-test-slave-qk05"
server_key: "0df74fe2824067186dbdc155ea07aa30"
server_key_iv: "cc1c19f7e2470377358c81c4a88e6482"
server_key_version: "encryptionkey@0"
I20260929 11:27:01.789448 28900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:01.796128 28900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:01.797123 28900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:01.797374 28900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:01.798319 28900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:01.799865 28900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:01.799953 28900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:01.800040 28900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:01.800091 28900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:01.834208 28900 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:46581
I20260929 11:27:01.834255 29026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:46581 every 8 connection(s)
I20260929 11:27:01.835688 28900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:27:01.838351 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 28900
I20260929 11:27:01.838650 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:27:01.839521 24164 external_mini_cluster.cc:1428] Setting key 27dd65c8a86a4d324797eb7fc02d801a
I20260929 11:27:01.842823 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42533
--builtin_ntp_servers=127.23.153.20:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 11:27:01.846304 29029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:01.839389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:34123 (local address 127.23.153.62:42533)
0929 11:27:01.839871 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:01.839879 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:01.841591 (+ 1712us) server_negotiation.cc:407] Connection header received
0929 11:27:01.843515 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:01.843521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:01.843594 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:01.843693 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:01.845276 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:01.845281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:01.845285 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:01.845304 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:01.845346 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:01.845359 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:01.845764 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:01.846008 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 11:27:01.846073 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":236,"threads_started":1}
I20260929 11:27:01.847736 29030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:01.840188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42533 (local address 127.23.153.1:34123)
0929 11:27:01.841409 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:01.841462 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:01.843189 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:01.843992 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:01.844002 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:01.844121 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:01.844883 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:27:01.844886 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:01.844960 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:27:01.845072 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:01.845688 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:01.845700 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:01.845711 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 11:27:01.846058 (+ 347us) client_negotiation.cc:240] Negotiation successful
0929 11:27:01.846387 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":345,"threads_started":1}
I20260929 11:27:01.849651 29027 heartbeater.cc:347] Connected to a master server at 127.23.153.62:42533
I20260929 11:27:01.850287 29027 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:01.851096 29027 heartbeater.cc:511] Master 127.23.153.62:42533 requested a full tablet report, sending...
I20260929 11:27:01.853792 28841 ts_manager.cc:194] Registered new tserver with Master: 0ad1aba11e9a4a41a2f99fc71d5ab0a7 (127.23.153.1:46581)
W20260929 11:27:02.181349 29031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:02.181664 29031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:02.181756 29031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:02.189340 29031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:02.189472 29031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:02.189586 29031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:27:02.199182 29031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:02.200495 29031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:02.202317 29031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:02.216755 29036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:02.216727 29039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:02.216948 29037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:02.217592 29031 server_base.cc:1034] running on GCE node
I20260929 11:27:02.218274 29031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:02.219485 29031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:02.220736 29031 hybrid_clock.cc:648] HybridClock initialized: now 1790681222220683 us; error 46 us; skew 500 ppm
I20260929 11:27:02.221097 29031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:27:02.223709 29031 webserver.cc:460] Webserver started at http://127.23.153.2:43539/ using document root <none> and password file <none>
I20260929 11:27:02.224498 29031 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:02.224613 29031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:02.224967 29031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:02.228240 29031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "0cfe72775b1140088d67e6d03a55a41e"
format_stamp: "Formatted at 2026-09-29 11:27:02 on dist-test-slave-qk05"
server_key: "54ea57d0185a7e96d9c08a2980d7d70a"
server_key_iv: "90a2de3faa18462cd028e4d53700c461"
server_key_version: "encryptionkey@0"
I20260929 11:27:02.229168 29031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "0cfe72775b1140088d67e6d03a55a41e"
format_stamp: "Formatted at 2026-09-29 11:27:02 on dist-test-slave-qk05"
server_key: "54ea57d0185a7e96d9c08a2980d7d70a"
server_key_iv: "90a2de3faa18462cd028e4d53700c461"
server_key_version: "encryptionkey@0"
I20260929 11:27:02.235437 29031 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 11:27:02.239542 29045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:02.241180 29031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:27:02.241389 29031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "0cfe72775b1140088d67e6d03a55a41e"
format_stamp: "Formatted at 2026-09-29 11:27:02 on dist-test-slave-qk05"
server_key: "54ea57d0185a7e96d9c08a2980d7d70a"
server_key_iv: "90a2de3faa18462cd028e4d53700c461"
server_key_version: "encryptionkey@0"
I20260929 11:27:02.241636 29031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:02.248030 29031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:02.248961 29031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:02.249246 29031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:02.250319 29031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:02.252084 29031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:02.252198 29031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:02.252316 29031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:02.252385 29031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:02.285085 29031 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:38527
I20260929 11:27:02.285099 29157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:38527 every 8 connection(s)
I20260929 11:27:02.286856 29031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:27:02.296612 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 29031
I20260929 11:27:02.296674 29029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:02.290282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:38299 (local address 127.23.153.62:42533)
0929 11:27:02.290562 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:02.290574 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:02.292482 (+ 1908us) server_negotiation.cc:407] Connection header received
0929 11:27:02.294305 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:02.294310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:02.294401 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:02.294503 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:02.295937 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:02.295942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:02.295945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:02.295963 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:02.295994 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:02.296006 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:02.296105 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:02.296395 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 11:27:02.296454 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260929 11:27:02.296797 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:27:02.297361 24164 external_mini_cluster.cc:1428] Setting key 7ec07dfa327054bcf3eaa003aafdfd20
I20260929 11:27:02.298090 29160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:02.290982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42533 (local address 127.23.153.2:38299)
0929 11:27:02.292155 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:02.292222 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:02.293997 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:02.294800 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:02.294812 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:02.294925 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:02.295599 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:27:02.295602 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:02.295645 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:27:02.295745 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:02.296124 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:02.296135 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:02.296146 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 11:27:02.296469 (+ 323us) client_negotiation.cc:240] Negotiation successful
0929 11:27:02.296805 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":335,"threads_started":1}
I20260929 11:27:02.299517 29158 heartbeater.cc:347] Connected to a master server at 127.23.153.62:42533
I20260929 11:27:02.299847 29158 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:02.300236 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42533
--builtin_ntp_servers=127.23.153.20:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 11:27:02.300486 29158 heartbeater.cc:511] Master 127.23.153.62:42533 requested a full tablet report, sending...
I20260929 11:27:02.302191 28841 ts_manager.cc:194] Registered new tserver with Master: 0cfe72775b1140088d67e6d03a55a41e (127.23.153.2:38527)
W20260929 11:27:02.639896 29161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:02.640214 29161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:02.640309 29161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:02.648177 29161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:02.648330 29161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:02.648483 29161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:27:02.658223 29161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:02.659632 29161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:02.661468 29161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:02.675499 29167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:02.675639 29166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:02.675546 29169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:02.676668 29161 server_base.cc:1034] running on GCE node
I20260929 11:27:02.677322 29161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:02.678350 29161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:02.679581 29161 hybrid_clock.cc:648] HybridClock initialized: now 1790681222679526 us; error 50 us; skew 500 ppm
I20260929 11:27:02.679943 29161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:27:02.682466 29161 webserver.cc:460] Webserver started at http://127.23.153.3:44017/ using document root <none> and password file <none>
I20260929 11:27:02.683320 29161 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:02.683439 29161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:02.683765 29161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:02.686825 29161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "25524d3e27cb43ad859f8cb60990f45a"
format_stamp: "Formatted at 2026-09-29 11:27:02 on dist-test-slave-qk05"
server_key: "42460b44b0c2ee5f7f1b6e25fc249791"
server_key_iv: "9cfb530fb44ea8c829ec9bbedca32c57"
server_key_version: "encryptionkey@0"
I20260929 11:27:02.687783 29161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "25524d3e27cb43ad859f8cb60990f45a"
format_stamp: "Formatted at 2026-09-29 11:27:02 on dist-test-slave-qk05"
server_key: "42460b44b0c2ee5f7f1b6e25fc249791"
server_key_iv: "9cfb530fb44ea8c829ec9bbedca32c57"
server_key_version: "encryptionkey@0"
I20260929 11:27:02.694049 29161 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:27:02.698104 29175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:02.699714 29161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:27:02.699911 29161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "25524d3e27cb43ad859f8cb60990f45a"
format_stamp: "Formatted at 2026-09-29 11:27:02 on dist-test-slave-qk05"
server_key: "42460b44b0c2ee5f7f1b6e25fc249791"
server_key_iv: "9cfb530fb44ea8c829ec9bbedca32c57"
server_key_version: "encryptionkey@0"
I20260929 11:27:02.700139 29161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:02.708981 29161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:02.710112 29161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:02.710398 29161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:02.711627 29161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:02.713696 29161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:02.713833 29161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:02.713956 29161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:02.714032 29161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:02.750805 29161 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:39789
I20260929 11:27:02.750852 29287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:39789 every 8 connection(s)
I20260929 11:27:02.752213 29161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:27:02.760910 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 29161
I20260929 11:27:02.761121 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:27:02.761428 29029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:02.754973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:37313 (local address 127.23.153.62:42533)
0929 11:27:02.755169 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:02.755174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:02.756852 (+ 1678us) server_negotiation.cc:407] Connection header received
0929 11:27:02.758661 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:02.758667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:02.758752 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:02.758840 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:02.760510 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:02.760518 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:02.760523 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:02.760552 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:02.760610 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:02.760631 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:02.760812 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:02.761012 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 11:27:02.761095 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 11:27:02.761734 24164 external_mini_cluster.cc:1428] Setting key 686c216e9ae8c4755531440fd60ebdbb
I20260929 11:27:02.762532 29290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:02.755629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42533 (local address 127.23.153.3:37313)
0929 11:27:02.756666 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:02.756714 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:02.758324 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:02.759177 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:02.759187 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:02.759301 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:02.759929 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:27:02.759931 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:02.759974 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:27:02.760101 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:02.760780 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:02.760790 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:02.760797 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 11:27:02.761071 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 11:27:02.761364 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":299,"threads_started":1}
I20260929 11:27:02.764225 29288 heartbeater.cc:347] Connected to a master server at 127.23.153.62:42533
I20260929 11:27:02.764559 29288 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:02.765308 29288 heartbeater.cc:511] Master 127.23.153.62:42533 requested a full tablet report, sending...
I20260929 11:27:02.766821 28841 ts_manager.cc:194] Registered new tserver with Master: 25524d3e27cb43ad859f8cb60990f45a (127.23.153.3:39789)
I20260929 11:27:02.778108 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:27:02.785753 29029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:02.782686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33118 (local address 127.23.153.62:42533)
0929 11:27:02.782864 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:02.782870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:02.783371 (+ 501us) server_negotiation.cc:407] Connection header received
0929 11:27:02.783554 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:02.783558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:02.783613 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:02.783699 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:02.785035 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:02.785042 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:02.785047 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:02.785068 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:02.785106 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:02.785120 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:02.785282 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:02.785392 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 11:27:02.785460 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 11:27:02.790941 28841 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:27:02.798229 28841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:27:02.832329 29301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:02.812043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48850 (local address 127.23.153.3:39789)
0929 11:27:02.827568 (+ 15525us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:02.827580 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:02.827636 (+ 56us) server_negotiation.cc:407] Connection header received
0929 11:27:02.827740 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:02.827750 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:02.828060 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:02.828282 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:02.831101 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:02.831113 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:02.831131 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:02.831155 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:02.831323 (+ 168us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:02.831350 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:02.831540 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:02.831808 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 11:27:02.831922 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15170,"thread_start_us":15022,"threads_started":1}
I20260929 11:27:02.832559 29300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:02.811983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:39789 (local address 127.0.0.1:48850)
0929 11:27:02.813211 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:02.813256 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:02.813388 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:02.828510 (+ 15122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:02.828525 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:02.828711 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:02.830695 (+ 1984us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:27:02.830698 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:02.830736 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:27:02.830837 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:02.831556 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:02.831580 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:02.831590 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 11:27:02.831816 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 11:27:02.831932 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":597,"threads_started":1}
I20260929 11:27:02.835203 29305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:02.815009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53922 (local address 127.23.153.2:38527)
0929 11:27:02.820710 (+ 5701us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:02.820978 (+ 268us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:02.831255 (+ 10277us) server_negotiation.cc:407] Connection header received
0929 11:27:02.831344 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:02.831352 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:02.831638 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:02.831796 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:02.834188 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:02.834200 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:02.834216 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:02.834249 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:02.834317 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:02.834342 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:02.834459 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:02.834722 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 11:27:02.834855 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4992,"thread_start_us":4826,"threads_started":1}
I20260929 11:27:02.836079 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:02.814356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:38527 (local address 127.0.0.1:53922)
0929 11:27:02.830610 (+ 16254us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:02.830641 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:02.830791 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:02.833010 (+ 2219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:02.833014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:02.833035 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:02.833820 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:27:02.833826 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:02.833844 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:27:02.833923 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:02.834557 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:02.834562 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:02.834566 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 11:27:02.835575 (+ 1009us) client_negotiation.cc:240] Negotiation successful
0929 11:27:02.835644 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16063,"thread_start_us":11422,"threads_started":1}
I20260929 11:27:02.837607 29223 tablet_service.cc:1467] Processing CreateTablet for tablet 823cdbffe432459681a5e873782c9690 (DEFAULT_TABLE table=test-table [id=022c3d78a4db473f984c317b00454483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:02.840551 29223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 823cdbffe432459681a5e873782c9690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:02.843892 29093 tablet_service.cc:1467] Processing CreateTablet for tablet 823cdbffe432459681a5e873782c9690 (DEFAULT_TABLE table=test-table [id=022c3d78a4db473f984c317b00454483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:02.845288 29093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 823cdbffe432459681a5e873782c9690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:02.852942 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:02.813703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:46581 (local address 127.0.0.1:34690)
0929 11:27:02.825452 (+ 11749us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:02.825487 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:02.825672 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:02.832107 (+ 6435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:02.832113 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:02.832144 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:02.833036 (+ 892us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:27:02.833039 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:02.833058 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:27:02.833126 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:02.851969 (+ 18843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:02.851976 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:02.851983 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 11:27:02.852146 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 11:27:02.852217 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11499,"thread_start_us":16416,"threads_started":1}
I20260929 11:27:02.853785 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:02.813913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.23.153.1:46581)
0929 11:27:02.831288 (+ 17375us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:02.831429 (+ 141us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:02.831549 (+ 120us) server_negotiation.cc:407] Connection header received
0929 11:27:02.831628 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:02.831648 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:02.831864 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:02.833388 (+ 1524us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:02.851644 (+ 18256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:02.851654 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:02.851670 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:02.851689 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:02.851748 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:02.851769 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:02.853309 (+ 1540us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:02.853393 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 11:27:02.853518 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17229,"thread_start_us":17113,"threads_started":1}
I20260929 11:27:02.857897 29027 heartbeater.cc:503] Master 127.23.153.62:42533 was elected leader, sending a full tablet report...
I20260929 11:27:02.858361 28962 tablet_service.cc:1467] Processing CreateTablet for tablet 823cdbffe432459681a5e873782c9690 (DEFAULT_TABLE table=test-table [id=022c3d78a4db473f984c317b00454483]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:02.860040 28962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 823cdbffe432459681a5e873782c9690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:02.860838 29306 tablet_bootstrap.cc:492] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a: Bootstrap starting.
I20260929 11:27:02.863505 29307 tablet_bootstrap.cc:492] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e: Bootstrap starting.
I20260929 11:27:02.864880 29306 tablet_bootstrap.cc:654] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:02.866672 29306 log.cc:824] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:02.868050 29307 tablet_bootstrap.cc:654] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:02.870033 29307 log.cc:824] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:02.871626 29306 tablet_bootstrap.cc:492] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a: No bootstrap required, opened a new log
I20260929 11:27:02.872103 29306 ts_tablet_manager.cc:1397] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 11:27:02.875168 29307 tablet_bootstrap.cc:492] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e: No bootstrap required, opened a new log
I20260929 11:27:02.875505 29307 ts_tablet_manager.cc:1397] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260929 11:27:02.876132 29310 tablet_bootstrap.cc:492] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7: Bootstrap starting.
I20260929 11:27:02.879413 29310 tablet_bootstrap.cc:654] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:02.881203 29310 log.cc:824] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:02.882803 29306 raft_consensus.cc:348] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25524d3e27cb43ad859f8cb60990f45a" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39789 } } peers { permanent_uuid: "0ad1aba11e9a4a41a2f99fc71d5ab0a7" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 46581 } } peers { permanent_uuid: "0cfe72775b1140088d67e6d03a55a41e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38527 } }
I20260929 11:27:02.883325 29306 raft_consensus.cc:374] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:02.883428 29306 raft_consensus.cc:729] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25524d3e27cb43ad859f8cb60990f45a, State: Initialized, Role: FOLLOWER
I20260929 11:27:02.884296 29310 tablet_bootstrap.cc:492] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7: No bootstrap required, opened a new log
I20260929 11:27:02.884152 29306 consensus_queue.cc:260] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25524d3e27cb43ad859f8cb60990f45a" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39789 } } peers { permanent_uuid: "0ad1aba11e9a4a41a2f99fc71d5ab0a7" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 46581 } } peers { permanent_uuid: "0cfe72775b1140088d67e6d03a55a41e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38527 } }
I20260929 11:27:02.884603 29310 ts_tablet_manager.cc:1397] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 11:27:02.886375 29307 raft_consensus.cc:348] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25524d3e27cb43ad859f8cb60990f45a" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39789 } } peers { permanent_uuid: "0ad1aba11e9a4a41a2f99fc71d5ab0a7" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 46581 } } peers { permanent_uuid: "0cfe72775b1140088d67e6d03a55a41e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38527 } }
I20260929 11:27:02.886948 29307 raft_consensus.cc:374] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:02.887108 29307 raft_consensus.cc:729] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfe72775b1140088d67e6d03a55a41e, State: Initialized, Role: FOLLOWER
I20260929 11:27:02.887933 29307 consensus_queue.cc:260] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25524d3e27cb43ad859f8cb60990f45a" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39789 } } peers { permanent_uuid: "0ad1aba11e9a4a41a2f99fc71d5ab0a7" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 46581 } } peers { permanent_uuid: "0cfe72775b1140088d67e6d03a55a41e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38527 } }
I20260929 11:27:02.889281 29158 heartbeater.cc:503] Master 127.23.153.62:42533 was elected leader, sending a full tablet report...
I20260929 11:27:02.890028 29307 ts_tablet_manager.cc:1428] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20260929 11:27:02.892912 29310 raft_consensus.cc:348] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25524d3e27cb43ad859f8cb60990f45a" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39789 } } peers { permanent_uuid: "0ad1aba11e9a4a41a2f99fc71d5ab0a7" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 46581 } } peers { permanent_uuid: "0cfe72775b1140088d67e6d03a55a41e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38527 } }
I20260929 11:27:02.893456 29310 raft_consensus.cc:374] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:02.893573 29310 raft_consensus.cc:729] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ad1aba11e9a4a41a2f99fc71d5ab0a7, State: Initialized, Role: FOLLOWER
I20260929 11:27:02.894260 29310 consensus_queue.cc:260] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25524d3e27cb43ad859f8cb60990f45a" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39789 } } peers { permanent_uuid: "0ad1aba11e9a4a41a2f99fc71d5ab0a7" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 46581 } } peers { permanent_uuid: "0cfe72775b1140088d67e6d03a55a41e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38527 } }
I20260929 11:27:02.895855 29288 heartbeater.cc:503] Master 127.23.153.62:42533 was elected leader, sending a full tablet report...
I20260929 11:27:02.896270 29306 ts_tablet_manager.cc:1428] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a: Time spent starting tablet: real 0.024s user 0.017s sys 0.004s
I20260929 11:27:02.896332 29310 ts_tablet_manager.cc:1428] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260929 11:27:02.955667 29313 raft_consensus.cc:482] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:27:02.955997 29313 raft_consensus.cc:504] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25524d3e27cb43ad859f8cb60990f45a" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39789 } } peers { permanent_uuid: "0ad1aba11e9a4a41a2f99fc71d5ab0a7" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 46581 } } peers { permanent_uuid: "0cfe72775b1140088d67e6d03a55a41e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38527 } }
I20260929 11:27:02.957672 29313 leader_election.cc:288] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25524d3e27cb43ad859f8cb60990f45a (127.23.153.3:39789), 0ad1aba11e9a4a41a2f99fc71d5ab0a7 (127.23.153.1:46581)
I20260929 11:27:02.961599 29301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:02.958173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:52365 (local address 127.23.153.3:39789)
0929 11:27:02.958546 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:02.958553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:02.959137 (+ 584us) server_negotiation.cc:407] Connection header received
0929 11:27:02.959407 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:02.959414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:02.959488 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:02.959606 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:02.960851 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:02.960857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:02.960862 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:02.960883 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:02.960917 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:02.960931 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:02.961051 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:02.961175 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 11:27:02.961246 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 11:27:02.961807 29315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:02.958108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:39789 (local address 127.23.153.2:52365)
0929 11:27:02.958959 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:02.958985 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:02.959181 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:02.959746 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:02.959750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:02.959770 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:02.960550 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:27:02.960553 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:02.960568 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:27:02.960634 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:02.961038 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:02.961043 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:02.961051 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 11:27:02.961226 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 11:27:02.961349 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":562,"threads_started":1}
I20260929 11:27:02.961897 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:02.959460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:37673 (local address 127.23.153.1:46581)
0929 11:27:02.959640 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:02.959645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:02.959909 (+ 264us) server_negotiation.cc:407] Connection header received
0929 11:27:02.960006 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:02.960012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:02.960071 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:02.960170 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:02.961299 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:02.961304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:02.961307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:02.961324 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:02.961355 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:02.961366 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:02.961475 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:02.961572 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 11:27:02.961633 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 11:27:02.962213 29316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:02.959166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:46581 (local address 127.23.153.2:37673)
0929 11:27:02.959800 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:02.959815 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:02.959913 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:02.960215 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:02.960218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:02.960239 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:02.961096 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:27:02.961099 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:02.961112 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:27:02.961149 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:02.961455 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:02.961460 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:02.961465 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 11:27:02.961602 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 11:27:02.961656 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":303,"threads_started":1}
I20260929 11:27:02.962957 29243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823cdbffe432459681a5e873782c9690" candidate_uuid: "0cfe72775b1140088d67e6d03a55a41e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25524d3e27cb43ad859f8cb60990f45a" is_pre_election: true
I20260929 11:27:02.963300 28982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823cdbffe432459681a5e873782c9690" candidate_uuid: "0cfe72775b1140088d67e6d03a55a41e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ad1aba11e9a4a41a2f99fc71d5ab0a7" is_pre_election: true
I20260929 11:27:02.963514 29243 raft_consensus.cc:2454] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfe72775b1140088d67e6d03a55a41e in term 0.
I20260929 11:27:02.963778 28982 raft_consensus.cc:2454] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfe72775b1140088d67e6d03a55a41e in term 0.
I20260929 11:27:02.964249 29049 leader_election.cc:302] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfe72775b1140088d67e6d03a55a41e, 25524d3e27cb43ad859f8cb60990f45a; no voters:
I20260929 11:27:02.964665 29313 raft_consensus.cc:2781] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:27:02.964823 29313 raft_consensus.cc:482] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:27:02.964931 29313 raft_consensus.cc:3037] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:02.967476 29313 raft_consensus.cc:504] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25524d3e27cb43ad859f8cb60990f45a" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39789 } } peers { permanent_uuid: "0ad1aba11e9a4a41a2f99fc71d5ab0a7" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 46581 } } peers { permanent_uuid: "0cfe72775b1140088d67e6d03a55a41e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38527 } }
I20260929 11:27:02.968140 29313 leader_election.cc:288] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [CANDIDATE]: Term 1 election: Requested vote from peers 25524d3e27cb43ad859f8cb60990f45a (127.23.153.3:39789), 0ad1aba11e9a4a41a2f99fc71d5ab0a7 (127.23.153.1:46581)
I20260929 11:27:02.968822 29243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823cdbffe432459681a5e873782c9690" candidate_uuid: "0cfe72775b1140088d67e6d03a55a41e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25524d3e27cb43ad859f8cb60990f45a"
I20260929 11:27:02.968869 28982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823cdbffe432459681a5e873782c9690" candidate_uuid: "0cfe72775b1140088d67e6d03a55a41e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ad1aba11e9a4a41a2f99fc71d5ab0a7"
I20260929 11:27:02.969048 29243 raft_consensus.cc:3037] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:02.969063 28982 raft_consensus.cc:3037] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:02.971503 28982 raft_consensus.cc:2454] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfe72775b1140088d67e6d03a55a41e in term 1.
I20260929 11:27:02.971549 29243 raft_consensus.cc:2454] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfe72775b1140088d67e6d03a55a41e in term 1.
I20260929 11:27:02.972039 29049 leader_election.cc:302] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfe72775b1140088d67e6d03a55a41e, 25524d3e27cb43ad859f8cb60990f45a; no voters:
I20260929 11:27:02.972391 29313 raft_consensus.cc:2781] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:02.972844 29313 raft_consensus.cc:686] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [term 1 LEADER]: Becoming Leader. State: Replica: 0cfe72775b1140088d67e6d03a55a41e, State: Running, Role: LEADER
I20260929 11:27:02.973420 29313 consensus_queue.cc:237] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25524d3e27cb43ad859f8cb60990f45a" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39789 } } peers { permanent_uuid: "0ad1aba11e9a4a41a2f99fc71d5ab0a7" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 46581 } } peers { permanent_uuid: "0cfe72775b1140088d67e6d03a55a41e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38527 } }
I20260929 11:27:02.978415 28841 catalog_manager.cc:5495] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfe72775b1140088d67e6d03a55a41e (127.23.153.2). New cstate: current_term: 1 leader_uuid: "0cfe72775b1140088d67e6d03a55a41e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25524d3e27cb43ad859f8cb60990f45a" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ad1aba11e9a4a41a2f99fc71d5ab0a7" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 46581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfe72775b1140088d67e6d03a55a41e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38527 } health_report { overall_health: HEALTHY } } }
W20260929 11:27:03.003973 29289 tablet.cc:2369] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:27:03.006120 29305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:03.003980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53930 (local address 127.23.153.2:38527)
0929 11:27:03.004147 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:03.004153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:03.004175 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:27:03.004354 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:03.004359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:03.004447 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:03.004542 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:03.005474 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:03.005479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:03.005482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:03.005500 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:03.005533 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:03.005546 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:03.005654 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:03.005762 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 11:27:03.005826 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:27:03.015188 29243 raft_consensus.cc:1261] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a [term 1 FOLLOWER]: Refusing update from remote peer 0cfe72775b1140088d67e6d03a55a41e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:27:03.015161 28982 raft_consensus.cc:1261] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7 [term 1 FOLLOWER]: Refusing update from remote peer 0cfe72775b1140088d67e6d03a55a41e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:27:03.016286 29317 consensus_queue.cc:1035] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ad1aba11e9a4a41a2f99fc71d5ab0a7" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 46581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:03.016680 29313 consensus_queue.cc:1035] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "25524d3e27cb43ad859f8cb60990f45a" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:03.042745 28841 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:27:03.043072 28841 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:27:03.046829 28841 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b139c75c38747f5b8f4997da3254bbf: Sending DeleteTablet for 3 replicas of tablet 823cdbffe432459681a5e873782c9690
I20260929 11:27:03.048226 29223 tablet_service.cc:1514] Processing DeleteTablet for tablet 823cdbffe432459681a5e873782c9690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:03 UTC) from {username='slave'} at 127.0.0.1:48850
I20260929 11:27:03.048334 29093 tablet_service.cc:1514] Processing DeleteTablet for tablet 823cdbffe432459681a5e873782c9690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:03 UTC) from {username='slave'} at 127.0.0.1:53922
I20260929 11:27:03.049125 29330 tablet_replica.cc:330] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a: stopping tablet replica
I20260929 11:27:03.049077 28962 tablet_service.cc:1514] Processing DeleteTablet for tablet 823cdbffe432459681a5e873782c9690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:03 UTC) from {username='slave'} at 127.0.0.1:34690
I20260929 11:27:03.049686 29330 raft_consensus.cc:2229] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:27:03.050283 29330 raft_consensus.cc:2258] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:03.052266 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 28900
I20260929 11:27:03.052842 29330 ts_tablet_manager.cc:1905] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:03.053906 29332 tablet_replica.cc:330] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7: stopping tablet replica
I20260929 11:27:03.054352 29332 raft_consensus.cc:2229] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:27:03.054865 29332 raft_consensus.cc:2258] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:03.055106 29331 tablet_replica.cc:330] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e: stopping tablet replica
I20260929 11:27:03.055538 29331 raft_consensus.cc:2229] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:27:03.056206 29331 raft_consensus.cc:2258] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:03.058748 29332 ts_tablet_manager.cc:1905] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:03.060657 29330 ts_tablet_manager.cc:1918] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:27:03.060942 29330 log.cc:1196] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/823cdbffe432459681a5e873782c9690
I20260929 11:27:03.061558 29330 ts_tablet_manager.cc:1939] T 823cdbffe432459681a5e873782c9690 P 25524d3e27cb43ad859f8cb60990f45a: Deleting consensus metadata
I20260929 11:27:03.063776 28828 catalog_manager.cc:4841] TS 25524d3e27cb43ad859f8cb60990f45a (127.23.153.3:39789): tablet 823cdbffe432459681a5e873782c9690 (table test-table [id=022c3d78a4db473f984c317b00454483]) successfully deleted
I20260929 11:27:03.065052 29331 ts_tablet_manager.cc:1905] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:03.066465 29303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:03.053617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34692 (local address 127.23.153.1:46581)
0929 11:27:03.053826 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:03.053832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:03.063641 (+ 9809us) server_negotiation.cc:407] Connection header received
0929 11:27:03.064153 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:03.064160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:03.064263 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:03.064342 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:03.065567 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:03.065575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:03.065579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:03.065601 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:03.065647 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:03.065664 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:03.065832 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:03.066064 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 11:27:03.066134 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 11:27:03.067885 29022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:03.073436 29331 ts_tablet_manager.cc:1918] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:27:03.073685 29331 log.cc:1196] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/823cdbffe432459681a5e873782c9690
I20260929 11:27:03.074272 29331 ts_tablet_manager.cc:1939] T 823cdbffe432459681a5e873782c9690 P 0cfe72775b1140088d67e6d03a55a41e: Deleting consensus metadata
I20260929 11:27:03.076328 28826 catalog_manager.cc:4841] TS 0cfe72775b1140088d67e6d03a55a41e (127.23.153.2:38527): tablet 823cdbffe432459681a5e873782c9690 (table test-table [id=022c3d78a4db473f984c317b00454483]) successfully deleted
I20260929 11:27:03.158846 29332 ts_tablet_manager.cc:1918] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:27:03.159124 29332 log.cc:1196] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/823cdbffe432459681a5e873782c9690
I20260929 11:27:03.159621 29332 ts_tablet_manager.cc:1939] T 823cdbffe432459681a5e873782c9690 P 0ad1aba11e9a4a41a2f99fc71d5ab0a7: Deleting consensus metadata
I20260929 11:27:03.159709 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 28900
W20260929 11:27:03.183771 28828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.153.1:46581 (error 108)
I20260929 11:27:03.184036 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 29031
I20260929 11:27:03.185998 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:03.185473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:46581 (local address 127.0.0.1:34704)
0929 11:27:03.185662 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:03.185815 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:46581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 11:27:03.186311 28828 catalog_manager.cc:4584] TS 0ad1aba11e9a4a41a2f99fc71d5ab0a7 (127.23.153.1:46581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 823cdbffe432459681a5e873782c9690: Network error: Client connection negotiation failed: client connection to 127.23.153.1:46581: connect: Connection refused (error 111)
I20260929 11:27:03.188256 29305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:03.185113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53932 (local address 127.23.153.2:38527)
0929 11:27:03.185305 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:03.185312 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:03.185336 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:27:03.186278 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:03.186285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:03.186366 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:03.186495 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:03.187587 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:03.187594 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:03.187599 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:03.187625 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:03.187666 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:03.187684 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:03.187823 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:03.187881 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 11:27:03.187947 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:27:03.189123 29153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:03.240084 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:03.239615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:46581 (local address 127.0.0.1:34712)
0929 11:27:03.239831 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:03.239919 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:46581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 11:27:03.241042 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:03.240703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:46581 (local address 127.0.0.1:34726)
0929 11:27:03.240844 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:03.240925 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:46581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 11:27:03.301846 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:03.301347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:46581 (local address 127.0.0.1:34734)
0929 11:27:03.301526 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:03.301629 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:46581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 11:27:03.303123 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 29031
I20260929 11:27:03.303606 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:03.303161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:46581 (local address 127.0.0.1:34740)
0929 11:27:03.303386 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:03.303469 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:46581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260929 11:27:03.320994 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 29161
I20260929 11:27:03.324054 29301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:03.321775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.23.153.3:39789)
0929 11:27:03.322031 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:03.322037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:03.322080 (+ 43us) server_negotiation.cc:407] Connection header received
0929 11:27:03.322268 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:03.322273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:03.322360 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:03.322449 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:03.323404 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:03.323409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:03.323413 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:03.323430 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:03.323466 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:03.323478 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:03.323604 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:03.323681 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 11:27:03.323757 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:27:03.325079 29283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:03.384462 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:03.383978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:46581 (local address 127.0.0.1:34752)
0929 11:27:03.384223 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:03.384317 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:46581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 11:27:03.385512 29304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:03.385106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:46581 (local address 127.0.0.1:34766)
0929 11:27:03.385313 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:03.385384 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:46581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 11:27:03.415261 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 29161
I20260929 11:27:03.429862 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 28810
I20260929 11:27:03.431244 28871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:03.540421 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 28810
2026-09-29T11:27:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2792 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 11:27:03.579934 24164 mini_oidc.cc:152] Starting JWKS server
I20260929 11:27:03.583254 24164 webserver.cc:460] Webserver started at https://127.0.0.1:37111/ using document root <none> and password file <none>
I20260929 11:27:03.583626 24164 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 11:27:03.584632 24164 webserver.cc:460] Webserver started at http://127.0.0.1:37525/ using document root <none> and password file <none>
2026-09-29T11:27:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:27:03Z Disabled control of system clock
I20260929 11:27:03.611776 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:40033
--webserver_interface=127.23.153.62
--webserver_port=0
--builtin_ntp_servers=127.23.153.20:38635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:40033
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37111/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/testchainca.cert with env {}
W20260929 11:27:03.983683 29344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:03.984007 29344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:03.984099 29344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:03.992256 29344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:27:03.992398 29344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:03.992439 29344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:03.992497 29344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:27:03.992542 29344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 11:27:03.992607 29344 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 11:27:03.992648 29344 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37111/jwks.json
I20260929 11:27:04.003602 29344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:38635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:40033
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:40033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37111/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:04.004979 29344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:04.006299 29344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:04.019394 29349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:04.019338 29352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:04.019369 29350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:04.020493 29344 server_base.cc:1034] running on GCE node
I20260929 11:27:04.021351 29344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:04.023042 29344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:04.024276 29344 hybrid_clock.cc:648] HybridClock initialized: now 1790681224024228 us; error 76 us; skew 500 ppm
I20260929 11:27:04.024628 29344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:27:04.027207 29344 webserver.cc:460] Webserver started at http://127.23.153.62:42543/ using document root <none> and password file <none>
I20260929 11:27:04.027954 29344 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:04.028057 29344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:04.028339 29344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:04.031471 29344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "7abe0dec7e3f4d7683a5c8c43bd236fc"
format_stamp: "Formatted at 2026-09-29 11:27:04 on dist-test-slave-qk05"
server_key: "347adde61967379d489baafaecf7a647"
server_key_iv: "3453e12190c19b32768fe35677e8854b"
server_key_version: "encryptionkey@0"
I20260929 11:27:04.032400 29344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "7abe0dec7e3f4d7683a5c8c43bd236fc"
format_stamp: "Formatted at 2026-09-29 11:27:04 on dist-test-slave-qk05"
server_key: "347adde61967379d489baafaecf7a647"
server_key_iv: "3453e12190c19b32768fe35677e8854b"
server_key_version: "encryptionkey@0"
I20260929 11:27:04.038774 29344 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260929 11:27:04.042856 29358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:04.044652 29344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:27:04.044878 29344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "7abe0dec7e3f4d7683a5c8c43bd236fc"
format_stamp: "Formatted at 2026-09-29 11:27:04 on dist-test-slave-qk05"
server_key: "347adde61967379d489baafaecf7a647"
server_key_iv: "3453e12190c19b32768fe35677e8854b"
server_key_version: "encryptionkey@0"
I20260929 11:27:04.045101 29344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:04.060012 29344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:04.060878 29344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:04.061159 29344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:04.080960 29344 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:40033
I20260929 11:27:04.080977 29409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:40033 every 8 connection(s)
I20260929 11:27:04.082408 29344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:27:04.083963 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 29344
I20260929 11:27:04.084218 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:27:04.084853 24164 external_mini_cluster.cc:1428] Setting key 1e50f7cc334d1db762b180d0c6dd8c6d
I20260929 11:27:04.087594 29410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:04.100330 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:04.086833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45160 (local address 127.23.153.62:40033)
0929 11:27:04.088184 (+ 1351us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:04.088326 (+ 142us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:04.088427 (+ 101us) server_negotiation.cc:407] Connection header received
0929 11:27:04.090992 (+ 2565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:04.091068 (+ 76us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:04.091708 (+ 640us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:04.092154 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:04.092792 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:04.094214 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:04.095670 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:04.096388 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:04.096600 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:04.097800 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:04.097808 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:04.097826 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:04.098467 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:04.098567 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:04.098597 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:04.098706 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:04.098941 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 11:27:04.099315 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":862,"thread_start_us":612,"threads_started":1}
I20260929 11:27:04.100371 29410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc: Bootstrap starting.
I20260929 11:27:04.103754 29410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:04.104291 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:27:04.104991 29410 log.cc:824] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:04.108294 29410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc: No bootstrap required, opened a new log
I20260929 11:27:04.116551 29410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7abe0dec7e3f4d7683a5c8c43bd236fc" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40033 } }
I20260929 11:27:04.116991 29410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:04.117096 29410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7abe0dec7e3f4d7683a5c8c43bd236fc, State: Initialized, Role: FOLLOWER
I20260929 11:27:04.117854 29410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7abe0dec7e3f4d7683a5c8c43bd236fc" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40033 } }
I20260929 11:27:04.118124 29410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:27:04.118355 29410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:27:04.118558 29410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:04.120965 29410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7abe0dec7e3f4d7683a5c8c43bd236fc" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40033 } }
I20260929 11:27:04.122184 29410 leader_election.cc:302] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7abe0dec7e3f4d7683a5c8c43bd236fc; no voters:
I20260929 11:27:04.123111 29410 leader_election.cc:288] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:27:04.123364 29415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:04.124076 29415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc [term 1 LEADER]: Becoming Leader. State: Replica: 7abe0dec7e3f4d7683a5c8c43bd236fc, State: Running, Role: LEADER
I20260929 11:27:04.124832 29415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7abe0dec7e3f4d7683a5c8c43bd236fc" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40033 } }
I20260929 11:27:04.125344 29410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:27:04.127848 29416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7abe0dec7e3f4d7683a5c8c43bd236fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7abe0dec7e3f4d7683a5c8c43bd236fc" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40033 } } }
I20260929 11:27:04.128321 29416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:04.128806 29417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7abe0dec7e3f4d7683a5c8c43bd236fc. Latest consensus state: current_term: 1 leader_uuid: "7abe0dec7e3f4d7683a5c8c43bd236fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7abe0dec7e3f4d7683a5c8c43bd236fc" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40033 } } }
I20260929 11:27:04.129146 29417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:04.129613 29423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:27:04.136152 29423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:27:04.146660 29423 catalog_manager.cc:1347] Generated new cluster ID: 395d82dbb4c5423797a554f291f1a573
I20260929 11:27:04.146793 29423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:27:04.168808 29423 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:27:04.170214 29423 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:27:04.179772 29423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7abe0dec7e3f4d7683a5c8c43bd236fc: Generated new TSK 0
I20260929 11:27:04.180951 29423 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 11:27:04.225271 29440 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 11:27:04.225473 29440 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 11:27:04.226580 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:04.203408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45166 (local address 127.23.153.62:40033)
0929 11:27:04.203718 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:04.203726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:04.204227 (+ 501us) server_negotiation.cc:407] Connection header received
0929 11:27:04.204650 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:04.204654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:04.204763 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:04.204905 (+ 142us) server_negotiation.cc:226] Negotiated authn=JWT
0929 11:27:04.205581 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:04.206413 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:04.208596 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:04.209109 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:04.209310 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:04.209918 (+ 608us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 11:27:04.225864 (+ 15946us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 11:27:04.226232 (+ 368us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":120}
W20260929 11:27:04.226715 29413 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45166: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 11:27:04.228776 29439 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:04.203363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:40033 (local address 127.0.0.1:45166)
0929 11:27:04.204087 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:04.204107 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:04.204345 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:04.204961 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:04.204967 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:04.204982 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0929 11:27:04.205395 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:04.205408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:04.206561 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:04.206567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:04.208385 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:04.208404 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:04.209309 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:04.209317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:04.209507 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:04.209603 (+ 96us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 11:27:04.226334 (+ 16731us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 11:27:04.226826 (+ 492us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.153.62:40033: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":342,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.23.153.62:40033: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 11:27:04.230780 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 29344
I20260929 11:27:04.232112 29405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:04.328035 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 29344
2026-09-29T11:27:04Z chronyd exiting
I20260929 11:27:04.377243 24164 test_util.cc:175] -----------------------------------------------
I20260929 11:27:04.377408 24164 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790681181154781-24164-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (801 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 11:27:04.381372 24164 mini_oidc.cc:152] Starting JWKS server
I20260929 11:27:04.383253 24164 webserver.cc:460] Webserver started at https://127.0.0.1:33043/ using document root <none> and password file <none>
I20260929 11:27:04.383466 24164 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 11:27:04.384124 24164 webserver.cc:460] Webserver started at http://127.0.0.1:33883/ using document root <none> and password file <none>
2026-09-29T11:27:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:27:04Z Disabled control of system clock
I20260929 11:27:04.413340 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:45973
--webserver_interface=127.23.153.62
--webserver_port=0
--builtin_ntp_servers=127.23.153.20:45531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:45973
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33043/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/testchainca.cert with env {}
W20260929 11:27:04.762166 29447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:04.762483 29447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:04.762600 29447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:04.770493 29447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:27:04.770627 29447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:04.770666 29447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:04.770733 29447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:27:04.770790 29447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 11:27:04.770848 29447 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 11:27:04.770885 29447 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33043/jwks.json
I20260929 11:27:04.782145 29447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:45531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:45973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:45973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33043/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:04.783542 29447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:04.784945 29447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:04.796540 29453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:04.796568 29455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:04.796609 29452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:04.797732 29447 server_base.cc:1034] running on GCE node
I20260929 11:27:04.798564 29447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:04.800089 29447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:04.801309 29447 hybrid_clock.cc:648] HybridClock initialized: now 1790681224801263 us; error 42 us; skew 500 ppm
I20260929 11:27:04.801664 29447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:27:04.804030 29447 webserver.cc:460] Webserver started at http://127.23.153.62:36325/ using document root <none> and password file <none>
I20260929 11:27:04.804756 29447 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:04.804868 29447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:04.805191 29447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:04.808158 29447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "c8db0ef15b3144f7aee6e7a83b9528dc"
format_stamp: "Formatted at 2026-09-29 11:27:04 on dist-test-slave-qk05"
server_key: "3ad64813be17714a36ae48940267e7d2"
server_key_iv: "4750e2faf5f13e467d259d76cd357aee"
server_key_version: "encryptionkey@0"
I20260929 11:27:04.808965 29447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "c8db0ef15b3144f7aee6e7a83b9528dc"
format_stamp: "Formatted at 2026-09-29 11:27:04 on dist-test-slave-qk05"
server_key: "3ad64813be17714a36ae48940267e7d2"
server_key_iv: "4750e2faf5f13e467d259d76cd357aee"
server_key_version: "encryptionkey@0"
I20260929 11:27:04.814518 29447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 11:27:04.818526 29461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:04.820209 29447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:27:04.820569 29447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "c8db0ef15b3144f7aee6e7a83b9528dc"
format_stamp: "Formatted at 2026-09-29 11:27:04 on dist-test-slave-qk05"
server_key: "3ad64813be17714a36ae48940267e7d2"
server_key_iv: "4750e2faf5f13e467d259d76cd357aee"
server_key_version: "encryptionkey@0"
I20260929 11:27:04.820845 29447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:04.847496 29447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:04.848413 29447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:04.848726 29447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:04.870340 29512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:45973 every 8 connection(s)
I20260929 11:27:04.870424 29447 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:45973
I20260929 11:27:04.872252 29447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:27:04.874394 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 29447
I20260929 11:27:04.874598 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:27:04.875432 24164 external_mini_cluster.cc:1428] Setting key 10fc6239943d5b601c8462be284dcdf8
I20260929 11:27:04.877463 29513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:04.889514 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:04.877686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.23.153.62:45973)
0929 11:27:04.878711 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:04.878723 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:04.878766 (+ 43us) server_negotiation.cc:407] Connection header received
0929 11:27:04.880364 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:04.880387 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:04.880915 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:04.881312 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:04.881898 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:04.883330 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:04.884693 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:04.885414 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:04.885590 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:04.886979 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:04.886987 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:04.887008 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:04.887710 (+ 702us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:04.887800 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:04.887825 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:04.887943 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:04.888191 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 11:27:04.888519 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":345,"threads_started":1}
I20260929 11:27:04.889696 29513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc: Bootstrap starting.
I20260929 11:27:04.893445 29513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:04.893816 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:27:04.894533 29513 log.cc:824] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:04.897645 29513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc: No bootstrap required, opened a new log
I20260929 11:27:04.905167 29513 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8db0ef15b3144f7aee6e7a83b9528dc" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45973 } }
I20260929 11:27:04.905591 29513 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:04.905685 29513 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8db0ef15b3144f7aee6e7a83b9528dc, State: Initialized, Role: FOLLOWER
I20260929 11:27:04.906286 29513 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8db0ef15b3144f7aee6e7a83b9528dc" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45973 } }
I20260929 11:27:04.906599 29513 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:27:04.906806 29513 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:27:04.906992 29513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:04.909389 29513 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8db0ef15b3144f7aee6e7a83b9528dc" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45973 } }
I20260929 11:27:04.910023 29513 leader_election.cc:302] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8db0ef15b3144f7aee6e7a83b9528dc; no voters:
I20260929 11:27:04.910652 29513 leader_election.cc:288] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:27:04.910820 29518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:04.911556 29518 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc [term 1 LEADER]: Becoming Leader. State: Replica: c8db0ef15b3144f7aee6e7a83b9528dc, State: Running, Role: LEADER
I20260929 11:27:04.912109 29518 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8db0ef15b3144f7aee6e7a83b9528dc" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45973 } }
I20260929 11:27:04.912814 29513 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:27:04.915195 29520 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8db0ef15b3144f7aee6e7a83b9528dc. Latest consensus state: current_term: 1 leader_uuid: "c8db0ef15b3144f7aee6e7a83b9528dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8db0ef15b3144f7aee6e7a83b9528dc" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45973 } } }
I20260929 11:27:04.915565 29520 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:04.915104 29519 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8db0ef15b3144f7aee6e7a83b9528dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8db0ef15b3144f7aee6e7a83b9528dc" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45973 } } }
I20260929 11:27:04.915692 29519 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:04.920140 29527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:27:04.925892 29527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:27:04.936064 29527 catalog_manager.cc:1347] Generated new cluster ID: 0038c473df374110977cbac872ab027f
I20260929 11:27:04.936196 29527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:27:04.959069 29527 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:27:04.960994 29527 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:27:04.974773 29527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8db0ef15b3144f7aee6e7a83b9528dc: Generated new TSK 0
I20260929 11:27:04.976024 29527 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 11:27:05.020020 29543 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 11:27:05.020165 29543 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 11:27:05.021132 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:04.992228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42740 (local address 127.23.153.62:45973)
0929 11:27:04.992440 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:04.992447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:04.992992 (+ 545us) server_negotiation.cc:407] Connection header received
0929 11:27:04.993486 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:04.993493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:04.993634 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:04.993941 (+ 307us) server_negotiation.cc:226] Negotiated authn=JWT
0929 11:27:04.994525 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:04.995565 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:04.997876 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:04.998378 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:04.998510 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:04.999255 (+ 745us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 11:27:05.020556 (+ 21301us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 11:27:05.020866 (+ 310us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42740: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 11:27:05.021503 29542 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:04.992212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:45973 (local address 127.0.0.1:42740)
0929 11:27:04.992835 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:04.992856 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:04.993047 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:04.993904 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:04.993910 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:04.993925 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0929 11:27:04.994277 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:04.994291 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:04.995738 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:04.995744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:04.997465 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:04.997486 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:04.998594 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:04.998602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:04.998808 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:04.998863 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 11:27:05.020811 (+ 21948us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 11:27:05.021106 (+ 295us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.153.62:45973: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":320,"threads_started":1}
W20260929 11:27:05.021519 29516 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42740: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260929 11:27:05.022848 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 29447
I20260929 11:27:05.024474 29508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:05.122422 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 29447
2026-09-29T11:27:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (794 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 11:27:05.176578 24164 mini_oidc.cc:152] Starting JWKS server
I20260929 11:27:05.178553 24164 webserver.cc:460] Webserver started at https://127.0.0.1:41081/ using document root <none> and password file <none>
I20260929 11:27:05.178807 24164 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 11:27:05.179543 24164 webserver.cc:460] Webserver started at http://127.0.0.1:46075/ using document root <none> and password file <none>
2026-09-29T11:27:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:27:05Z Disabled control of system clock
I20260929 11:27:05.207374 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:37855
--webserver_interface=127.23.153.62
--webserver_port=0
--builtin_ntp_servers=127.23.153.20:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:37855
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41081/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 11:27:05.575901 29550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:05.576256 29550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:05.576344 29550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:05.584456 29550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:27:05.584604 29550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:05.584656 29550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:05.584713 29550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:27:05.584757 29550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 11:27:05.584816 29550 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 11:27:05.584856 29550 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41081/jwks.json
I20260929 11:27:05.594892 29550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:37855
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:37855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41081/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:05.596380 29550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:05.598286 29550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:05.610530 29556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:05.610617 29558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:05.610563 29555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:05.611469 29550 server_base.cc:1034] running on GCE node
I20260929 11:27:05.612442 29550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:05.614022 29550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:05.615263 29550 hybrid_clock.cc:648] HybridClock initialized: now 1790681225615213 us; error 51 us; skew 500 ppm
I20260929 11:27:05.615665 29550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:27:05.618147 29550 webserver.cc:460] Webserver started at http://127.23.153.62:38271/ using document root <none> and password file <none>
I20260929 11:27:05.618955 29550 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:05.619107 29550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:05.619429 29550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:05.622576 29550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "c2d46ed8d7e74760b00200c3b9f5d797"
format_stamp: "Formatted at 2026-09-29 11:27:05 on dist-test-slave-qk05"
server_key: "713cd721a4cdb84fda75af81397191c9"
server_key_iv: "2aeb80a24de8326cd5f1376f3448bc0e"
server_key_version: "encryptionkey@0"
I20260929 11:27:05.623490 29550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "c2d46ed8d7e74760b00200c3b9f5d797"
format_stamp: "Formatted at 2026-09-29 11:27:05 on dist-test-slave-qk05"
server_key: "713cd721a4cdb84fda75af81397191c9"
server_key_iv: "2aeb80a24de8326cd5f1376f3448bc0e"
server_key_version: "encryptionkey@0"
I20260929 11:27:05.629703 29550 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:27:05.634552 29564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:05.636513 29550 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:27:05.636766 29550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "c2d46ed8d7e74760b00200c3b9f5d797"
format_stamp: "Formatted at 2026-09-29 11:27:05 on dist-test-slave-qk05"
server_key: "713cd721a4cdb84fda75af81397191c9"
server_key_iv: "2aeb80a24de8326cd5f1376f3448bc0e"
server_key_version: "encryptionkey@0"
I20260929 11:27:05.637020 29550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:05.660405 29550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:05.661325 29550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:05.661631 29550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:05.682250 29550 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:37855
I20260929 11:27:05.682289 29615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:37855 every 8 connection(s)
I20260929 11:27:05.684239 29550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:27:05.688506 29616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:05.690654 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 29550
I20260929 11:27:05.690889 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:27:05.691689 24164 external_mini_cluster.cc:1428] Setting key 5b16fd0b8ee79265f05f85ab135bbbe3
I20260929 11:27:05.706892 29616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797: Bootstrap starting.
I20260929 11:27:05.708796 29619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:05.694270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39232 (local address 127.23.153.62:37855)
0929 11:27:05.695516 (+ 1246us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:05.695531 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:05.695580 (+ 49us) server_negotiation.cc:407] Connection header received
0929 11:27:05.697222 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:05.697250 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:05.697824 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:05.698251 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:05.698830 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:05.700278 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:05.701724 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:05.702513 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:05.702717 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:05.704538 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:05.704547 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:05.704566 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:05.705416 (+ 850us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:05.705547 (+ 131us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:05.705589 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:05.705844 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:05.706224 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 11:27:05.706953 (+ 729us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":419,"threads_started":1}
I20260929 11:27:05.711251 29616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:05.712548 29616 log.cc:824] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:05.714357 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:27:05.716029 29616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797: No bootstrap required, opened a new log
I20260929 11:27:05.724781 29616 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d46ed8d7e74760b00200c3b9f5d797" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37855 } }
I20260929 11:27:05.725216 29616 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:05.725314 29616 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d46ed8d7e74760b00200c3b9f5d797, State: Initialized, Role: FOLLOWER
I20260929 11:27:05.725942 29616 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d46ed8d7e74760b00200c3b9f5d797" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37855 } }
I20260929 11:27:05.726169 29616 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:27:05.726300 29616 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:27:05.726451 29616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:05.728977 29616 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d46ed8d7e74760b00200c3b9f5d797" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37855 } }
I20260929 11:27:05.729605 29616 leader_election.cc:302] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2d46ed8d7e74760b00200c3b9f5d797; no voters:
I20260929 11:27:05.730250 29616 leader_election.cc:288] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:27:05.730396 29621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:05.731076 29621 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797 [term 1 LEADER]: Becoming Leader. State: Replica: c2d46ed8d7e74760b00200c3b9f5d797, State: Running, Role: LEADER
I20260929 11:27:05.731624 29621 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d46ed8d7e74760b00200c3b9f5d797" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37855 } }
I20260929 11:27:05.732335 29616 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:27:05.734462 29622 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2d46ed8d7e74760b00200c3b9f5d797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d46ed8d7e74760b00200c3b9f5d797" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37855 } } }
I20260929 11:27:05.734894 29622 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:05.736033 29629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:27:05.735864 29623 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2d46ed8d7e74760b00200c3b9f5d797. Latest consensus state: current_term: 1 leader_uuid: "c2d46ed8d7e74760b00200c3b9f5d797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2d46ed8d7e74760b00200c3b9f5d797" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37855 } } }
I20260929 11:27:05.736205 29623 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:05.743077 29629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:27:05.754647 29629 catalog_manager.cc:1347] Generated new cluster ID: 24c2c272fce142e5a7f0cfa01867d36d
I20260929 11:27:05.754779 29629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:27:05.776336 29629 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:27:05.777828 29629 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:27:05.787278 29629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2d46ed8d7e74760b00200c3b9f5d797: Generated new TSK 0
I20260929 11:27:05.788267 29629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:27:05.813907 29619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:05.804192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39236 (local address 127.23.153.62:37855)
0929 11:27:05.804427 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:05.804434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:05.804873 (+ 439us) server_negotiation.cc:407] Connection header received
0929 11:27:05.805225 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:05.805230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:05.805320 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:05.805481 (+ 161us) server_negotiation.cc:226] Negotiated authn=JWT
0929 11:27:05.806048 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:05.806819 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:05.808915 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:05.809687 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:05.809952 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:05.810609 (+ 657us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 11:27:05.812826 (+ 2217us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41081/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 11:27:05.813294 (+ 468us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39236: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41081/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 11:27:05.814215 29619 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39236: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41081/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 11:27:05.814278 29645 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:05.803968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37855 (local address 127.0.0.1:39236)
0929 11:27:05.804743 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:05.804764 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:05.804950 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:05.805520 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:05.805525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:05.805539 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0929 11:27:05.805896 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:05.805907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:05.806973 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:05.806979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:05.808727 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:05.808747 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:05.810007 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:05.810014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:05.810233 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:05.810289 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 11:27:05.813090 (+ 2801us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41081/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 11:27:05.813475 (+ 385us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.153.62:37855: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41081/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":501,"threads_started":1}
I20260929 11:27:05.815980 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 29550
I20260929 11:27:05.817188 29611 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 11:27:05.828329 29646 webserver.cc:533] Webserver: sslize failed:
E20260929 11:27:05.828480 29646 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 11:27:05.920866 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 29550
2026-09-29T11:27:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (797 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T11:27:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:27:05Z Disabled control of system clock
I20260929 11:27:06.003955 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:34139
--webserver_interface=127.23.153.62
--webserver_port=0
--builtin_ntp_servers=127.23.153.20:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:34139
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/testchainca.cert with env {}
W20260929 11:27:06.363731 29651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:06.364069 29651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:06.364162 29651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:06.371783 29651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:27:06.371913 29651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:06.371958 29651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:06.372017 29651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:27:06.372061 29651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 11:27:06.372114 29651 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 11:27:06.372144 29651 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 11:27:06.383203 29651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:34139
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:34139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:06.384680 29651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:06.386053 29651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:06.398802 29656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:06.398732 29659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:06.398777 29657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:06.400178 29651 server_base.cc:1034] running on GCE node
I20260929 11:27:06.401168 29651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:06.403349 29651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:06.404695 29651 hybrid_clock.cc:648] HybridClock initialized: now 1790681226404633 us; error 70 us; skew 500 ppm
I20260929 11:27:06.405125 29651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:27:06.412264 29651 webserver.cc:460] Webserver started at http://127.23.153.62:46103/ using document root <none> and password file <none>
I20260929 11:27:06.413157 29651 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:06.413293 29651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:06.413617 29651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:06.417420 29651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "3bfabc5cf64145b1bc54f510025a7e4d"
format_stamp: "Formatted at 2026-09-29 11:27:06 on dist-test-slave-qk05"
server_key: "c5078843cf91044b7395a719cef1689e"
server_key_iv: "445668299d5e20e7605aedc4b3e05acf"
server_key_version: "encryptionkey@0"
I20260929 11:27:06.418711 29651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "3bfabc5cf64145b1bc54f510025a7e4d"
format_stamp: "Formatted at 2026-09-29 11:27:06 on dist-test-slave-qk05"
server_key: "c5078843cf91044b7395a719cef1689e"
server_key_iv: "445668299d5e20e7605aedc4b3e05acf"
server_key_version: "encryptionkey@0"
I20260929 11:27:06.426225 29651 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260929 11:27:06.432137 29665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:06.434485 29651 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:27:06.434768 29651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "3bfabc5cf64145b1bc54f510025a7e4d"
format_stamp: "Formatted at 2026-09-29 11:27:06 on dist-test-slave-qk05"
server_key: "c5078843cf91044b7395a719cef1689e"
server_key_iv: "445668299d5e20e7605aedc4b3e05acf"
server_key_version: "encryptionkey@0"
I20260929 11:27:06.435202 29651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:06.462488 29651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:06.463896 29651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:06.464255 29651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:06.484412 29651 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:34139
I20260929 11:27:06.484531 29716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:34139 every 8 connection(s)
I20260929 11:27:06.485924 29651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:27:06.486631 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 29651
I20260929 11:27:06.486812 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:27:06.487614 24164 external_mini_cluster.cc:1428] Setting key ef2da269e5bb2e6159bf8d33e4db42b4
I20260929 11:27:06.492233 29717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:06.504289 29720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:06.489659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60956 (local address 127.23.153.62:34139)
0929 11:27:06.490705 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:06.490726 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:06.490773 (+ 47us) server_negotiation.cc:407] Connection header received
0929 11:27:06.492586 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:06.492615 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:06.493422 (+ 807us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:06.494109 (+ 687us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:06.494844 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:06.496564 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:06.498915 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:06.499848 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:06.500043 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:06.501480 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:06.501489 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:06.501507 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:06.502128 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:06.502217 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:06.502244 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:06.502346 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:06.502703 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 11:27:06.503137 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":347,"threads_started":1}
I20260929 11:27:06.506942 29717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d: Bootstrap starting.
I20260929 11:27:06.508750 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:27:06.510970 29717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:06.512485 29717 log.cc:824] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:06.516065 29717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d: No bootstrap required, opened a new log
I20260929 11:27:06.525262 29717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfabc5cf64145b1bc54f510025a7e4d" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34139 } }
I20260929 11:27:06.525697 29717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:06.525804 29717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bfabc5cf64145b1bc54f510025a7e4d, State: Initialized, Role: FOLLOWER
I20260929 11:27:06.526484 29717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfabc5cf64145b1bc54f510025a7e4d" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34139 } }
I20260929 11:27:06.526772 29717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:27:06.526916 29717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:27:06.527138 29717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:06.529386 29717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfabc5cf64145b1bc54f510025a7e4d" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34139 } }
I20260929 11:27:06.530072 29717 leader_election.cc:302] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bfabc5cf64145b1bc54f510025a7e4d; no voters:
I20260929 11:27:06.530843 29717 leader_election.cc:288] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:27:06.531157 29722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:06.531811 29722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d [term 1 LEADER]: Becoming Leader. State: Replica: 3bfabc5cf64145b1bc54f510025a7e4d, State: Running, Role: LEADER
I20260929 11:27:06.532392 29722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfabc5cf64145b1bc54f510025a7e4d" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34139 } }
I20260929 11:27:06.533106 29717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:27:06.535501 29723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bfabc5cf64145b1bc54f510025a7e4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfabc5cf64145b1bc54f510025a7e4d" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34139 } } }
I20260929 11:27:06.535580 29724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bfabc5cf64145b1bc54f510025a7e4d. Latest consensus state: current_term: 1 leader_uuid: "3bfabc5cf64145b1bc54f510025a7e4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bfabc5cf64145b1bc54f510025a7e4d" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34139 } } }
I20260929 11:27:06.536028 29723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:06.536041 29724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:06.537284 29731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:27:06.544518 29731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:27:06.556227 29731 catalog_manager.cc:1347] Generated new cluster ID: 341ae8551e2f4543b65ef63613f18e09
I20260929 11:27:06.556358 29731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:27:06.570528 29731 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:27:06.572178 29731 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:27:06.580557 29731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bfabc5cf64145b1bc54f510025a7e4d: Generated new TSK 0
I20260929 11:27:06.581561 29731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:27:07.035818 29720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:06.607445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60972 (local address 127.23.153.62:34139)
0929 11:27:06.607613 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:06.607619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:06.608086 (+ 467us) server_negotiation.cc:407] Connection header received
0929 11:27:06.608333 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:06.608338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:06.608433 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:06.608594 (+ 161us) server_negotiation.cc:226] Negotiated authn=JWT
0929 11:27:06.609050 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:06.609767 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:06.611172 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:06.611625 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:06.611743 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:06.612254 (+ 511us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 11:27:07.034856 (+422602us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 11:27:07.035485 (+ 629us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60972: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 11:27:07.035953 29720 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60972: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 11:27:07.036092 29746 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:06.607380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:34139 (local address 127.0.0.1:60972)
0929 11:27:06.607971 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:06.607989 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:06.608149 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:06.608621 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:06.608626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:06.608637 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0929 11:27:06.608899 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:06.608908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:06.609922 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:06.609927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:06.610992 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:06.611003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:06.611809 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:06.611816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:06.611949 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:06.611986 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 11:27:07.035326 (+423340us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 11:27:07.035650 (+ 324us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.153.62:34139: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":308,"threads_started":1}
I20260929 11:27:07.037257 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 29651
I20260929 11:27:07.038425 29712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:07.148705 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 29651
2026-09-29T11:27:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1210 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (439 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (451 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:27:08 dist-test-slave-qk05 krb5kdc[29761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:27:08 dist-test-slave-qk05 krb5kdc[29761](info): set up 2 sockets
Sep 29 11:27:08 dist-test-slave-qk05 krb5kdc[29761](info): commencing operation
krb5kdc: starting...
W20260929 11:27:09.063922 24164 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.006s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:27:09 dist-test-slave-qk05 krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:27:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:27:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:27:09.277611 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:32943
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:32943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:09.617439 29774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:09.617762 29774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:09.617858 29774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:09.625569 29774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:27:09.625705 29774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:09.625765 29774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:09.625824 29774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:27:09.625869 29774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:27:09.636129 29774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:32943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:32943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:09.637470 29774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:09.638900 29774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:09.651430 29779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:09.651546 29782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:09.651404 29780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:09.652279 29774 server_base.cc:1034] running on GCE node
I20260929 11:27:09.653148 29774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:09.654772 29774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:09.656028 29774 hybrid_clock.cc:648] HybridClock initialized: now 1790681229655964 us; error 50 us; skew 500 ppm
I20260929 11:27:09.656420 29774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:09 dist-test-slave-qk05 krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681229, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:09.660004 29774 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:27:09.661782 29774 webserver.cc:460] Webserver started at http://127.23.153.62:35673/ using document root <none> and password file <none>
I20260929 11:27:09.662561 29774 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:09.662683 29774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:09.663062 29774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:09.666568 29774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "d117ab2c5e1d4700be1d837f4616859d"
format_stamp: "Formatted at 2026-09-29 11:27:09 on dist-test-slave-qk05"
server_key: "00ea78406de902a5ed8dc66a27f716c1"
server_key_iv: "33974c18f31bfa875fd66b3e861fe2c9"
server_key_version: "encryptionkey@0"
I20260929 11:27:09.667517 29774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "d117ab2c5e1d4700be1d837f4616859d"
format_stamp: "Formatted at 2026-09-29 11:27:09 on dist-test-slave-qk05"
server_key: "00ea78406de902a5ed8dc66a27f716c1"
server_key_iv: "33974c18f31bfa875fd66b3e861fe2c9"
server_key_version: "encryptionkey@0"
I20260929 11:27:09.673267 29774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:27:09.677345 29789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:09.679301 29774 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:27:09.679483 29774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "d117ab2c5e1d4700be1d837f4616859d"
format_stamp: "Formatted at 2026-09-29 11:27:09 on dist-test-slave-qk05"
server_key: "00ea78406de902a5ed8dc66a27f716c1"
server_key_iv: "33974c18f31bfa875fd66b3e861fe2c9"
server_key_version: "encryptionkey@0"
I20260929 11:27:09.679679 29774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:09.706225 29774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:09.711462 29774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:09.711733 29774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:09.730154 29774 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:32943
I20260929 11:27:09.730273 29840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:32943 every 8 connection(s)
I20260929 11:27:09.731791 29774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:27:09.736490 29841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:09.740947 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 29774
I20260929 11:27:09.741302 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:27:09.742063 24164 external_mini_cluster.cc:1428] Setting key 2ac0526a47c3288fc7a7ec400ddd3ceb
I20260929 11:27:09.748888 29841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d: Bootstrap starting.
I20260929 11:27:09.754026 29841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:09.755438 29841 log.cc:824] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d: Log is configured to *not* fsync() on all Append() calls
Sep 29 11:27:09 dist-test-slave-qk05 krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:27:09.759187 29841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d: No bootstrap required, opened a new log
I20260929 11:27:09.768637 29844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:09.744629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.23.153.62:32943)
0929 11:27:09.745535 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:09.745551 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:09.745596 (+ 45us) server_negotiation.cc:407] Connection header received
0929 11:27:09.747592 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:09.747620 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:09.748111 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:09.748538 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:09.750651 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:09.752280 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:09.754252 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:09.755108 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:09.755292 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:09.760115 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:09.760128 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:09.760147 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:09.760196 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:09.763735 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:09.764806 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:09.764815 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:09.764827 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:09.765051 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:09.765496 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:09.765501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:09.765507 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:09.766149 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:09.766463 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:09.766936 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 11:27:09.767475 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":419,"threads_started":1}
I20260929 11:27:09.768632 29841 raft_consensus.cc:348] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d117ab2c5e1d4700be1d837f4616859d" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 32943 } }
I20260929 11:27:09.769470 29841 raft_consensus.cc:374] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:09.769582 29841 raft_consensus.cc:729] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d117ab2c5e1d4700be1d837f4616859d, State: Initialized, Role: FOLLOWER
I20260929 11:27:09.770354 29841 consensus_queue.cc:260] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d117ab2c5e1d4700be1d837f4616859d" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 32943 } }
I20260929 11:27:09.770632 29841 raft_consensus.cc:388] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:27:09.770773 29841 raft_consensus.cc:482] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:27:09.770958 29841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:09.773669 29841 raft_consensus.cc:504] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d117ab2c5e1d4700be1d837f4616859d" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 32943 } }
I20260929 11:27:09.774387 29841 leader_election.cc:302] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d117ab2c5e1d4700be1d837f4616859d; no voters:
I20260929 11:27:09.774997 29841 leader_election.cc:288] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:27:09.775185 29846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:09.776369 29846 raft_consensus.cc:686] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d [term 1 LEADER]: Becoming Leader. State: Replica: d117ab2c5e1d4700be1d837f4616859d, State: Running, Role: LEADER
I20260929 11:27:09.777089 29846 consensus_queue.cc:237] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d117ab2c5e1d4700be1d837f4616859d" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 32943 } }
I20260929 11:27:09.777257 29841 sys_catalog.cc:564] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:27:09.780337 29848 sys_catalog.cc:455] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d117ab2c5e1d4700be1d837f4616859d. Latest consensus state: current_term: 1 leader_uuid: "d117ab2c5e1d4700be1d837f4616859d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d117ab2c5e1d4700be1d837f4616859d" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 32943 } } }
I20260929 11:27:09.780400 29847 sys_catalog.cc:455] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d117ab2c5e1d4700be1d837f4616859d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d117ab2c5e1d4700be1d837f4616859d" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 32943 } } }
I20260929 11:27:09.780778 29848 sys_catalog.cc:458] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:09.780786 29847 sys_catalog.cc:458] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:09.781711 29854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:27:09.792408 29854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:27:09.802369 29854 catalog_manager.cc:1347] Generated new cluster ID: 0471a22c5d8e40ac800bc23a084806be
I20260929 11:27:09.802495 29854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:27:09.826154 29854 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:27:09.828212 29854 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:27:09.838325 29854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d117ab2c5e1d4700be1d837f4616859d: Generated new TSK 0
I20260929 11:27:09.839311 29854 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
I20260929 11:27:09.899219 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:32943
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:10.258680 29867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:10.259042 29867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:10.259199 29867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:10.268424 29867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:10.268632 29867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:10.268808 29867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:27:10.280917 29867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:32943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:10.282269 29867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:10.283967 29867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:10.299343 29873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:10.299431 29875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:10.299400 29872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:10.300355 29867 server_base.cc:1034] running on GCE node
I20260929 11:27:10.301029 29867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:10.302205 29867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:10.303481 29867 hybrid_clock.cc:648] HybridClock initialized: now 1790681230303412 us; error 68 us; skew 500 ppm
I20260929 11:27:10.303866 29867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:10 dist-test-slave-qk05 krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681230, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:10.308291 29867 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:27:10.310124 29867 webserver.cc:460] Webserver started at http://127.23.153.1:38471/ using document root <none> and password file <none>
I20260929 11:27:10.310901 29867 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:10.311048 29867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:10.311769 29867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:10.315454 29867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "58a2639e77664d93be7d1adb68bff6ae"
format_stamp: "Formatted at 2026-09-29 11:27:10 on dist-test-slave-qk05"
server_key: "1c8d67351fe65efd8c2625137f066f61"
server_key_iv: "8ea472ce5214786cc36a690c4c3bd415"
server_key_version: "encryptionkey@0"
I20260929 11:27:10.316401 29867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "58a2639e77664d93be7d1adb68bff6ae"
format_stamp: "Formatted at 2026-09-29 11:27:10 on dist-test-slave-qk05"
server_key: "1c8d67351fe65efd8c2625137f066f61"
server_key_iv: "8ea472ce5214786cc36a690c4c3bd415"
server_key_version: "encryptionkey@0"
I20260929 11:27:10.322492 29867 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 11:27:10.328248 29882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:10.330551 29867 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:27:10.330789 29867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "58a2639e77664d93be7d1adb68bff6ae"
format_stamp: "Formatted at 2026-09-29 11:27:10 on dist-test-slave-qk05"
server_key: "1c8d67351fe65efd8c2625137f066f61"
server_key_iv: "8ea472ce5214786cc36a690c4c3bd415"
server_key_version: "encryptionkey@0"
I20260929 11:27:10.331096 29867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:10.355686 29867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:10.361449 29867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:10.361721 29867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:10.362775 29867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:10.364493 29867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:10.364588 29867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:10.364681 29867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:10.364730 29867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:10.400767 29867 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:36841
I20260929 11:27:10.400923 29994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:36841 every 8 connection(s)
I20260929 11:27:10.402215 29867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:27:10.406095 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 29867
I20260929 11:27:10.406286 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:27:10.407145 24164 external_mini_cluster.cc:1428] Setting key 36a74d1f35cc74d7a60c0f39552c454b
Sep 29 11:27:10 dist-test-slave-qk05 krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681230, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:27:10.429592 29997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:10.405396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:35803 (local address 127.23.153.62:32943)
0929 11:27:10.405994 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:10.406000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:10.407446 (+ 1446us) server_negotiation.cc:407] Connection header received
0929 11:27:10.410059 (+ 2613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:10.410064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:10.410151 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:10.410248 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:10.412998 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:10.413856 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:10.416546 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:10.417125 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:10.417301 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:10.421069 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:10.421076 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:10.421081 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:10.421124 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:10.424674 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:10.426103 (+ 1429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:10.426111 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:10.426115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:10.426333 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:10.427293 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:10.427300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:10.427304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:10.427739 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:10.427954 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:10.428922 (+ 968us) server_negotiation.cc:299] Negotiation successful
0929 11:27:10.429211 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":237,"threads_started":1}
WARNING: no policy specified for kudu/127.23.153.2@KRBTEST.COM; defaulting to no policy
I20260929 11:27:10.430608 29998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:10.406113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:32943 (local address 127.23.153.1:35803)
0929 11:27:10.407108 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:10.407177 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:10.409590 (+ 2413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:10.410614 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:10.410629 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:10.411579 (+ 950us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:10.412586 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:10.412624 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:10.414139 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:10.414145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:10.416278 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:10.416301 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:10.417401 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:10.417407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:10.417633 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:10.418344 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:10.418370 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:10.420731 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:10.425037 (+ 4306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:10.425049 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:10.425067 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:10.425693 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:10.426542 (+ 849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:10.426546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:10.426549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:10.427056 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:10.427967 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:10.427975 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:10.428398 (+ 423us) client_negotiation.cc:769] Sending connection context
0929 11:27:10.428697 (+ 299us) client_negotiation.cc:240] Negotiation successful
0929 11:27:10.429182 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":381,"threads_started":1}
I20260929 11:27:10.433100 29995 heartbeater.cc:347] Connected to a master server at 127.23.153.62:32943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.2@KRBTEST.COM" created.
I20260929 11:27:10.433498 29995 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:10.434459 29995 heartbeater.cc:511] Master 127.23.153.62:32943 requested a full tablet report, sending...
I20260929 11:27:10.437407 29806 ts_manager.cc:194] Registered new tserver with Master: 58a2639e77664d93be7d1adb68bff6ae (127.23.153.1:36841)
I20260929 11:27:10.439524 29806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:35803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
I20260929 11:27:10.463951 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:32943
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:10.821265 30001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:10.821604 30001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:10.821743 30001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:10.829779 30001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:10.829947 30001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:10.830106 30001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:27:10.841635 30001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:32943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:10.843140 30001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:10.844553 30001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:10.860853 30007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:10.860775 30009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:10.860808 30006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:10.862113 30001 server_base.cc:1034] running on GCE node
I20260929 11:27:10.862867 30001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:10.864127 30001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:10.865423 30001 hybrid_clock.cc:648] HybridClock initialized: now 1790681230865337 us; error 76 us; skew 500 ppm
I20260929 11:27:10.865839 30001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:10 dist-test-slave-qk05 krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681230, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:10.869875 30001 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:27:10.872097 30001 webserver.cc:460] Webserver started at http://127.23.153.2:36249/ using document root <none> and password file <none>
I20260929 11:27:10.873111 30001 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:10.873251 30001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:10.873613 30001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:10.878832 30001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "49ce4e3e111042c18f523cb222b76e9e"
format_stamp: "Formatted at 2026-09-29 11:27:10 on dist-test-slave-qk05"
server_key: "59520b941056245b4dcc047ddb6c0d54"
server_key_iv: "809507771a7e188cc009c999f96705ef"
server_key_version: "encryptionkey@0"
I20260929 11:27:10.879912 30001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "49ce4e3e111042c18f523cb222b76e9e"
format_stamp: "Formatted at 2026-09-29 11:27:10 on dist-test-slave-qk05"
server_key: "59520b941056245b4dcc047ddb6c0d54"
server_key_iv: "809507771a7e188cc009c999f96705ef"
server_key_version: "encryptionkey@0"
I20260929 11:27:10.887530 30001 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260929 11:27:10.892112 30016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:10.894095 30001 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:27:10.894361 30001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "49ce4e3e111042c18f523cb222b76e9e"
format_stamp: "Formatted at 2026-09-29 11:27:10 on dist-test-slave-qk05"
server_key: "59520b941056245b4dcc047ddb6c0d54"
server_key_iv: "809507771a7e188cc009c999f96705ef"
server_key_version: "encryptionkey@0"
I20260929 11:27:10.894618 30001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:10.924144 30001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:10.929922 30001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:10.930225 30001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:10.931353 30001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:10.932889 30001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:10.932969 30001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:10.933066 30001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:10.933110 30001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:10.966078 30001 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:44913
I20260929 11:27:10.966109 30128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:44913 every 8 connection(s)
I20260929 11:27:10.967501 30001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:27:10.969852 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 30001
I20260929 11:27:10.970008 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:27:10.970438 24164 external_mini_cluster.cc:1428] Setting key 737821be3a7c0e7167e62e57f146277e
Sep 29 11:27:10 dist-test-slave-qk05 krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681230, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.153.3@KRBTEST.COM; defaulting to no policy
I20260929 11:27:10.993368 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:10.970507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:53355 (local address 127.23.153.62:32943)
0929 11:27:10.971112 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:10.971118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:10.973013 (+ 1895us) server_negotiation.cc:407] Connection header received
0929 11:27:10.975940 (+ 2927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:10.975945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:10.976025 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:10.976128 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:10.978345 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:10.979518 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:10.981447 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:10.982123 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:10.982276 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:10.986088 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:10.986095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:10.986100 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:10.986147 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:10.989879 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:10.990856 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:10.990862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:10.990866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:10.991072 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:10.991516 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:10.991521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:10.991524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:10.991908 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:10.992069 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:10.992889 (+ 820us) server_negotiation.cc:299] Negotiation successful
0929 11:27:10.993083 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":320,"threads_started":1}
I20260929 11:27:10.994407 30132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:10.971363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:32943 (local address 127.23.153.2:53355)
0929 11:27:10.972644 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:10.972712 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:10.975448 (+ 2736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:10.976428 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:10.976443 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:10.977214 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:10.978117 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:10.978146 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:10.979764 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:10.979770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:10.981246 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:10.981268 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:10.982378 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:10.982384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:10.982610 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:10.983369 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:10.983393 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:10.985787 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:10.990150 (+ 4363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:10.990161 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:10.990183 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:10.990703 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:10.991188 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:10.991193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:10.991197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:10.991404 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:10.992095 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:10.992103 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:10.992483 (+ 380us) client_negotiation.cc:769] Sending connection context
0929 11:27:10.992783 (+ 300us) client_negotiation.cc:240] Negotiation successful
0929 11:27:10.993244 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":498,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.3@KRBTEST.COM" created.
I20260929 11:27:10.996261 30129 heartbeater.cc:347] Connected to a master server at 127.23.153.62:32943
I20260929 11:27:10.996598 30129 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:10.997344 30129 heartbeater.cc:511] Master 127.23.153.62:32943 requested a full tablet report, sending...
I20260929 11:27:10.999078 29806 ts_manager.cc:194] Registered new tserver with Master: 49ce4e3e111042c18f523cb222b76e9e (127.23.153.2:44913)
I20260929 11:27:11.000169 29806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:53355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
I20260929 11:27:11.018882 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:32943
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:11.358829 30135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:11.359205 30135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:11.359349 30135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:11.366918 30135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:11.367127 30135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:11.367319 30135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:27:11.377743 30135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:32943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:11.379154 30135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:11.380491 30135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:11.394454 30140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:11.394373 30143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:11.394436 30141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:11.395457 30135 server_base.cc:1034] running on GCE node
I20260929 11:27:11.396130 30135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:11.397181 30135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:11.398424 30135 hybrid_clock.cc:648] HybridClock initialized: now 1790681231398370 us; error 48 us; skew 500 ppm
I20260929 11:27:11.398876 30135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:11 dist-test-slave-qk05 krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681231, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:11.403091 30135 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:27:11.405095 30135 webserver.cc:460] Webserver started at http://127.23.153.3:40857/ using document root <none> and password file <none>
I20260929 11:27:11.406131 30135 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:11.406298 30135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:11.406672 30135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:11.410180 30135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "f90d2f6bde5b42ac8582f56013bc5a39"
format_stamp: "Formatted at 2026-09-29 11:27:11 on dist-test-slave-qk05"
server_key: "2b807ab885528a618b90fe3a5638222d"
server_key_iv: "21400a91066c2ba6eba9f6620752454b"
server_key_version: "encryptionkey@0"
I20260929 11:27:11.411077 30135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "f90d2f6bde5b42ac8582f56013bc5a39"
format_stamp: "Formatted at 2026-09-29 11:27:11 on dist-test-slave-qk05"
server_key: "2b807ab885528a618b90fe3a5638222d"
server_key_iv: "21400a91066c2ba6eba9f6620752454b"
server_key_version: "encryptionkey@0"
I20260929 11:27:11.417150 30135 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:27:11.421748 30150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:11.423535 30135 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:27:11.423735 30135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "f90d2f6bde5b42ac8582f56013bc5a39"
format_stamp: "Formatted at 2026-09-29 11:27:11 on dist-test-slave-qk05"
server_key: "2b807ab885528a618b90fe3a5638222d"
server_key_iv: "21400a91066c2ba6eba9f6620752454b"
server_key_version: "encryptionkey@0"
I20260929 11:27:11.423942 30135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:11.443272 29995 heartbeater.cc:503] Master 127.23.153.62:32943 was elected leader, sending a full tablet report...
I20260929 11:27:11.455050 30135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:11.459666 30135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:11.459928 30135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:11.460930 30135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:11.462366 30135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:11.462448 30135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:11.462558 30135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:11.462603 30135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:11.495683 30135 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:39393
I20260929 11:27:11.495736 30262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:39393 every 8 connection(s)
I20260929 11:27:11.497042 30135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:27:11.500285 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 30135
I20260929 11:27:11.500468 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:27:11.500911 24164 external_mini_cluster.cc:1428] Setting key 01aa5092af78a04ba1bad4107c120807
Sep 29 11:27:11 dist-test-slave-qk05 krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681231, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:27:11.520368 30265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:11.500058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:59239 (local address 127.23.153.62:32943)
0929 11:27:11.500603 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:11.500610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:11.501806 (+ 1196us) server_negotiation.cc:407] Connection header received
0929 11:27:11.503801 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:11.503807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:11.503894 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:11.503988 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:11.505913 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:11.506716 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:11.508855 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:11.509291 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:11.509416 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:11.513795 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:11.513801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:11.513805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:11.513847 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:11.517017 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:11.517870 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:11.517874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:11.517876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:11.518003 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:11.518521 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:11.518526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:11.518528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:11.518828 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:11.518954 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:11.519886 (+ 932us) server_negotiation.cc:299] Negotiation successful
0929 11:27:11.520088 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":259,"threads_started":1}
I20260929 11:27:11.521399 30266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:11.500735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:32943 (local address 127.23.153.3:59239)
0929 11:27:11.501646 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:11.501700 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:11.503511 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:11.504280 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:11.504290 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:11.504858 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:11.505620 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:11.505642 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:11.506878 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:11.506882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:11.508647 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:11.508666 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:11.509541 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:11.509549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:11.509795 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:11.510526 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:11.510552 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:11.513417 (+ 2865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:11.517201 (+ 3784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:11.517210 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:11.517228 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:11.517747 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:11.518145 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:11.518150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:11.518152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:11.518369 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:11.518979 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:11.518987 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:11.519426 (+ 439us) client_negotiation.cc:769] Sending connection context
0929 11:27:11.519725 (+ 299us) client_negotiation.cc:240] Negotiation successful
0929 11:27:11.520185 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":383,"threads_started":1}
I20260929 11:27:11.523072 30263 heartbeater.cc:347] Connected to a master server at 127.23.153.62:32943
I20260929 11:27:11.523418 30263 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:11.524151 30263 heartbeater.cc:511] Master 127.23.153.62:32943 requested a full tablet report, sending...
I20260929 11:27:11.525759 29806 ts_manager.cc:194] Registered new tserver with Master: f90d2f6bde5b42ac8582f56013bc5a39 (127.23.153.3:39393)
I20260929 11:27:11.526757 29806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:59239
I20260929 11:27:11.529420 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/test-admin.keytab.
Sep 29 11:27:11 dist-test-slave-qk05 krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:11.557547 24164 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 11:27:11.563099 30271 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:11.560447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53228 (local address 127.23.153.1:36841)
0929 11:27:11.561117 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:11.561125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:11.561154 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:27:11.561216 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:11.561222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:11.561430 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:11.561759 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:11.562619 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53228: BlockingRecv error: recv got EOF from 127.0.0.1:53228 (error 108)
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":374,"threads_started":1}
W20260929 11:27:11.563494 24164 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/krb5cc' of type FILE
Sep 29 11:27:11 dist-test-slave-qk05 krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:11.565217 24164 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 11:27:11 dist-test-slave-qk05 krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:11.566665 24164 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 11:27:11 dist-test-slave-qk05 krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:27:11.586243 30271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:11.568800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.23.153.1:36841)
0929 11:27:11.569131 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:11.569139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:11.569598 (+ 459us) server_negotiation.cc:407] Connection header received
0929 11:27:11.569892 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:11.569897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:11.569969 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:11.570062 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:11.571283 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:11.572294 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:11.573753 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:11.574433 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:11.574566 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:11.579555 (+ 4989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:11.579576 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:11.579594 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:11.579642 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:11.582656 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:11.583850 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:11.583859 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:11.583862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:11.584024 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:11.584459 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:11.584464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:11.584466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:11.584933 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:11.585190 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:11.585599 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 11:27:11.585890 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
W20260929 11:27:11.594980 30271 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:11.592860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53256 (local address 127.23.153.1:36841)
0929 11:27:11.593101 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:11.593107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:11.593286 (+ 179us) server_negotiation.cc:407] Connection header received
0929 11:27:11.593541 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:11.593547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:11.593644 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:11.593787 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:11.594666 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53256: BlockingRecv error: recv got EOF from 127.0.0.1:53256 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 11:27:11.595250 24164 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/krb5cc' of type FILE
Sep 29 11:27:11 dist-test-slave-qk05 krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:11.596861 24164 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 11:27:11 dist-test-slave-qk05 krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:11.598362 24164 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 11:27:11 dist-test-slave-qk05 krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:27:11.616140 30271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:11.600094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53262 (local address 127.23.153.1:36841)
0929 11:27:11.600324 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:11.600330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:11.600552 (+ 222us) server_negotiation.cc:407] Connection header received
0929 11:27:11.600838 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:11.600843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:11.600915 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:11.601001 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:11.602058 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:11.602918 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:11.604323 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:11.604822 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:11.604936 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:11.609030 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:11.609036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:11.609040 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:11.609100 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:11.612443 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:11.613455 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:11.613460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:11.613463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:11.613615 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:11.614155 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:11.614162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:11.614165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:11.614657 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:11.614961 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:11.615443 (+ 482us) server_negotiation.cc:299] Negotiation successful
0929 11:27:11.615735 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 11:27:11.621117 30271 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:11.619198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.23.153.1:36841)
0929 11:27:11.619370 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:11.619378 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:11.619674 (+ 296us) server_negotiation.cc:407] Connection header received
0929 11:27:11.619841 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:11.619846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:11.619924 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:11.620009 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:11.620868 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53276: BlockingRecv error: recv got EOF from 127.0.0.1:53276 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 11:27:11.621371 24164 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790681181154781-24164-0/krb5kdc/krb5cc' of type FILE
Sep 29 11:27:11 dist-test-slave-qk05 krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:11.622884 24164 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 11:27:11 dist-test-slave-qk05 krb5kdc[29761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:11.624287 24164 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 11:27:11 dist-test-slave-qk05 krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:27:11.642176 30271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:11.626059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.23.153.1:36841)
0929 11:27:11.626419 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:11.626427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:11.626837 (+ 410us) server_negotiation.cc:407] Connection header received
0929 11:27:11.626954 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:11.626958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:11.627100 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:11.627195 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:11.628798 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:11.629619 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:11.631431 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:11.632105 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:11.632391 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:11.636613 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:11.636634 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:11.636638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:11.636688 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:11.639355 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:11.640260 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:11.640265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:11.640267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:11.640452 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:11.640907 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:11.640910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:11.640913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:11.641213 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:11.641464 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:11.641697 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 11:27:11.641893 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260929 11:27:11.644538 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 29867
I20260929 11:27:11.660439 30271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:11.645597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.23.153.1:36841)
0929 11:27:11.645933 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:11.645941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:11.646229 (+ 288us) server_negotiation.cc:407] Connection header received
0929 11:27:11.646417 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:11.646424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:11.646515 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:11.646601 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:11.647873 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:11.648656 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:11.650507 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:11.650983 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:11.651158 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:11.653914 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:11.653921 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:11.653925 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:11.653980 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:11.657155 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:11.658129 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:11.658135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:11.658137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:11.658329 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:11.658867 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:11.658871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:11.658873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:11.659278 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:11.659519 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:11.659914 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 11:27:11.660122 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260929 11:27:11.661188 29990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:11.750866 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 29867
I20260929 11:27:11.764707 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 30001
Sep 29 11:27:11 dist-test-slave-qk05 krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.2@KRBTEST.COM
I20260929 11:27:11.783166 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:11.765694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.23.153.2:44913)
0929 11:27:11.766253 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:11.766261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:11.766288 (+ 27us) server_negotiation.cc:407] Connection header received
0929 11:27:11.766350 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:11.766356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:11.766588 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:11.766727 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:11.767873 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:11.769339 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:11.770812 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:11.771668 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:11.771805 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:11.776119 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:11.776136 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:11.776154 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:11.776197 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:11.779667 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:11.780666 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:11.780676 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:11.780678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:11.780826 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:11.781505 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:11.781509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:11.781511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:11.781991 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:11.782250 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:11.782565 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 11:27:11.782797 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":275,"threads_started":1}
I20260929 11:27:11.784893 30124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:11.869062 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 30001
I20260929 11:27:11.882827 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 30135
Sep 29 11:27:11 dist-test-slave-qk05 krb5kdc[29761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.3@KRBTEST.COM
I20260929 11:27:11.900358 30286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:11.883880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.23.153.3:39393)
0929 11:27:11.884546 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:11.884554 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:11.884585 (+ 31us) server_negotiation.cc:407] Connection header received
0929 11:27:11.884649 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:11.884656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:11.884910 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:11.885067 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:11.886222 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:11.887319 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:11.888657 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:11.889249 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:11.889378 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:11.893687 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:11.893704 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:11.893722 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:11.893771 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:11.897189 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:11.898069 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:11.898086 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:11.898088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:11.898232 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:11.898646 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:11.898650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:11.898652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:11.899238 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:11.899500 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:11.899752 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 11:27:11.899978 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":328,"threads_started":1}
I20260929 11:27:11.901865 30258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:11.987818 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 30135
I20260929 11:27:12.000962 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 29774
I20260929 11:27:12.002502 29836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:12.107357 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 29774
2026-09-29T11:27:12Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4068 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:27:12 dist-test-slave-qk05 krb5kdc[30291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:27:12 dist-test-slave-qk05 krb5kdc[30291](info): set up 2 sockets
Sep 29 11:27:12 dist-test-slave-qk05 krb5kdc[30291](info): commencing operation
krb5kdc: starting...
W20260929 11:27:13.135953 24164 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.007s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:27:13 dist-test-slave-qk05 krb5kdc[30291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:27:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:27:13Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.62.keytab.
Entry for principal oryx/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.62.keytab.
I20260929 11:27:13.362370 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:41979
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.62.keytab
--principal=oryx/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:41979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:13.731684 30304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:13.732033 30304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:13.732141 30304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:13.742954 30304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:27:13.743204 30304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:13.743299 30304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:13.743394 30304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:27:13.743459 30304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:27:13.760533 30304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:41979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.62.keytab
--principal=oryx/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:41979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:13.762751 30304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:13.765381 30304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:13.780086 30309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:13.780750 30310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:13.781989 30312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:13.785240 30304 server_base.cc:1034] running on GCE node
I20260929 11:27:13.786188 30304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:13.788343 30304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:13.789619 30304 hybrid_clock.cc:648] HybridClock initialized: now 1790681233789547 us; error 71 us; skew 500 ppm
I20260929 11:27:13.790056 30304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:13 dist-test-slave-qk05 krb5kdc[30291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681233, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:13.793931 30304 init.cc:377] Logged in from keytab as oryx/127.23.153.62@KRBTEST.COM (short username oryx)
I20260929 11:27:13.795989 30304 webserver.cc:460] Webserver started at http://127.23.153.62:36707/ using document root <none> and password file <none>
I20260929 11:27:13.796787 30304 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:13.796901 30304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:13.797174 30304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:13.800727 30304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "bd60a4cf233742daa64ac4c57ea49ea7"
format_stamp: "Formatted at 2026-09-29 11:27:13 on dist-test-slave-qk05"
server_key: "c706d38f47e666d01a60b5d6741f585b"
server_key_iv: "6a04f29f7e770e9a3c563a81ea094822"
server_key_version: "encryptionkey@0"
I20260929 11:27:13.801590 30304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "bd60a4cf233742daa64ac4c57ea49ea7"
format_stamp: "Formatted at 2026-09-29 11:27:13 on dist-test-slave-qk05"
server_key: "c706d38f47e666d01a60b5d6741f585b"
server_key_iv: "6a04f29f7e770e9a3c563a81ea094822"
server_key_version: "encryptionkey@0"
I20260929 11:27:13.808018 30304 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 11:27:13.812927 30319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:13.815001 30304 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260929 11:27:13.815260 30304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "bd60a4cf233742daa64ac4c57ea49ea7"
format_stamp: "Formatted at 2026-09-29 11:27:13 on dist-test-slave-qk05"
server_key: "c706d38f47e666d01a60b5d6741f585b"
server_key_iv: "6a04f29f7e770e9a3c563a81ea094822"
server_key_version: "encryptionkey@0"
I20260929 11:27:13.815474 30304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:13.857586 30304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:13.862102 30304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:13.862354 30304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:13.881819 30304 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:41979
I20260929 11:27:13.881834 30370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:41979 every 8 connection(s)
I20260929 11:27:13.883381 30304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:27:13.886014 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 30304
I20260929 11:27:13.886230 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:27:13.887223 24164 external_mini_cluster.cc:1428] Setting key ed2cf9a56dcc4cfa304a9ffc5e357271
I20260929 11:27:13.888227 30371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 11:27:13 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM
I20260929 11:27:13.904320 30371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7: Bootstrap starting.
I20260929 11:27:13.908376 30371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:13.909619 30371 log.cc:824] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:13.913213 30371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7: No bootstrap required, opened a new log
I20260929 11:27:13.913794 30374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:13.890470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50156 (local address 127.23.153.62:41979)
0929 11:27:13.891366 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:13.891379 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:13.891428 (+ 49us) server_negotiation.cc:407] Connection header received
0929 11:27:13.893113 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:13.893141 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:13.893732 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:13.894133 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:13.896093 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:13.897498 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:13.899299 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:13.900091 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:13.900268 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:13.905520 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:13.905533 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:13.905557 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:13.905613 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:13.909110 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:13.910134 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:13.910142 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:13.910149 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:13.910446 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:13.910971 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:13.910977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:13.910980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:13.911627 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:13.911868 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:13.912324 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 11:27:13.912708 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":439,"threads_started":1}
I20260929 11:27:13.921768 30371 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } }
I20260929 11:27:13.922190 30371 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:13.922302 30371 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd60a4cf233742daa64ac4c57ea49ea7, State: Initialized, Role: FOLLOWER
I20260929 11:27:13.922950 30371 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } }
I20260929 11:27:13.923290 30371 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:27:13.923434 30371 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:27:13.923594 30371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:13.925904 30371 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } }
I20260929 11:27:13.926548 30371 leader_election.cc:302] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd60a4cf233742daa64ac4c57ea49ea7; no voters:
I20260929 11:27:13.927299 30371 leader_election.cc:288] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:27:13.927426 30376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:13.928048 30376 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [term 1 LEADER]: Becoming Leader. State: Replica: bd60a4cf233742daa64ac4c57ea49ea7, State: Running, Role: LEADER
I20260929 11:27:13.928671 30376 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } }
I20260929 11:27:13.929770 30371 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:27:13.932442 30378 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd60a4cf233742daa64ac4c57ea49ea7. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } }
I20260929 11:27:13.932904 30378 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:13.933732 30377 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } }
I20260929 11:27:13.934017 30377 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:13.936165 30382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:27:13.941480 30382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:27:13.951149 30382 catalog_manager.cc:1347] Generated new cluster ID: fb7a3e2cc82d4b3ba2e460537a75d64b
I20260929 11:27:13.951268 30382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:27:13.962009 30382 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:27:13.963469 30382 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:27:13.974632 30382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7: Generated new TSK 0
I20260929 11:27:13.975996 30382 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.1.keytab.
Entry for principal oryx/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.1.keytab.
I20260929 11:27:14.038177 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:41979
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.1.keytab
--principal=oryx/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:14.417217 30397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:14.417543 30397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:14.417677 30397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:14.426240 30397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:14.426383 30397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:14.426544 30397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:27:14.437500 30397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.1.keytab
--principal=oryx/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:41979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:14.438881 30397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:14.440310 30397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:14.454243 30402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:14.454205 30405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:14.454173 30403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:14.454871 30397 server_base.cc:1034] running on GCE node
I20260929 11:27:14.455610 30397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:14.456689 30397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:14.457911 30397 hybrid_clock.cc:648] HybridClock initialized: now 1790681234457852 us; error 51 us; skew 500 ppm
I20260929 11:27:14.458235 30397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:14 dist-test-slave-qk05 krb5kdc[30291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681234, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:14.461686 30397 init.cc:377] Logged in from keytab as oryx/127.23.153.1@KRBTEST.COM (short username oryx)
I20260929 11:27:14.463320 30397 webserver.cc:460] Webserver started at http://127.23.153.1:35671/ using document root <none> and password file <none>
I20260929 11:27:14.464085 30397 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:14.464205 30397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:14.464488 30397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:14.467633 30397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "35ce55ccb12942328602182facf3ad15"
format_stamp: "Formatted at 2026-09-29 11:27:14 on dist-test-slave-qk05"
server_key: "1b9484b18dde9e024d594510b86c8c40"
server_key_iv: "68ae3195f099b42528d59c1743a42b71"
server_key_version: "encryptionkey@0"
I20260929 11:27:14.468482 30397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "35ce55ccb12942328602182facf3ad15"
format_stamp: "Formatted at 2026-09-29 11:27:14 on dist-test-slave-qk05"
server_key: "1b9484b18dde9e024d594510b86c8c40"
server_key_iv: "68ae3195f099b42528d59c1743a42b71"
server_key_version: "encryptionkey@0"
I20260929 11:27:14.474300 30397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:27:14.478117 30412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:14.479789 30397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:27:14.479969 30397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "35ce55ccb12942328602182facf3ad15"
format_stamp: "Formatted at 2026-09-29 11:27:14 on dist-test-slave-qk05"
server_key: "1b9484b18dde9e024d594510b86c8c40"
server_key_iv: "68ae3195f099b42528d59c1743a42b71"
server_key_version: "encryptionkey@0"
I20260929 11:27:14.480162 30397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:14.511575 30397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:14.515967 30397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:14.516228 30397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:14.517197 30397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:14.518551 30397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:14.518630 30397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:14.518730 30397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:14.518774 30397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:14.550333 30397 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:41733
I20260929 11:27:14.550377 30524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:41733 every 8 connection(s)
I20260929 11:27:14.551738 30397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:27:14.553988 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 30397
I20260929 11:27:14.554195 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:27:14.555001 24164 external_mini_cluster.cc:1428] Setting key 31beae9ba7f4b42867736f3a9246a66a
Sep 29 11:27:14 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681234, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.23.153.2@KRBTEST.COM; defaulting to no policy
I20260929 11:27:14.574366 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:14.554512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:34469 (local address 127.23.153.62:41979)
0929 11:27:14.554947 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:14.555078 (+ 131us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:14.556490 (+ 1412us) server_negotiation.cc:407] Connection header received
0929 11:27:14.558558 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:14.558563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:14.558649 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:14.558745 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:14.560516 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:14.561283 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:14.563576 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:14.564008 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:14.564131 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:14.567998 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:14.568004 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:14.568007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:14.568044 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:14.570655 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:14.571566 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:14.571571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:14.571573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:14.571706 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:14.572168 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:14.572172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:14.572174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:14.572470 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:14.572610 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:14.573440 (+ 830us) server_negotiation.cc:299] Negotiation successful
0929 11:27:14.573986 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":232,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.153.2@KRBTEST.COM" created.
I20260929 11:27:14.576004 30528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:14.555402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.1:34469)
0929 11:27:14.556311 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:14.556365 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:14.558249 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:14.559065 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:14.559075 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:14.559628 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:14.560250 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:14.560270 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:14.561590 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:14.561595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:14.563363 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:14.563379 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:14.564260 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:14.564267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:14.564522 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:14.565210 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:14.565233 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:14.567707 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:14.570850 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:14.570859 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:14.570875 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:14.571405 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:14.571820 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:14.571824 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:14.571826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:14.572036 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:14.572607 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:14.572616 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:14.573027 (+ 411us) client_negotiation.cc:769] Sending connection context
0929 11:27:14.573305 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 11:27:14.574208 (+ 903us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":364,"threads_started":1}
I20260929 11:27:14.578142 30525 heartbeater.cc:347] Connected to a master server at 127.23.153.62:41979
I20260929 11:27:14.578477 30525 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:14.579347 30525 heartbeater.cc:511] Master 127.23.153.62:41979 requested a full tablet report, sending...
I20260929 11:27:14.582130 30336 ts_manager.cc:194] Registered new tserver with Master: 35ce55ccb12942328602182facf3ad15 (127.23.153.1:41733)
I20260929 11:27:14.584223 30336 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:34469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.2.keytab.
Entry for principal oryx/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.2.keytab.
I20260929 11:27:14.602495 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:41979
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.2.keytab
--principal=oryx/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:14.971140 30531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:14.971463 30531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:14.971589 30531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:14.979748 30531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:14.979889 30531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:14.980043 30531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:27:14.990708 30531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.2.keytab
--principal=oryx/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:41979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:14.992113 30531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:14.993590 30531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:15.008001 30537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:15.008044 30536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:15.007967 30539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:15.008867 30531 server_base.cc:1034] running on GCE node
I20260929 11:27:15.009543 30531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:15.010586 30531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:15.011823 30531 hybrid_clock.cc:648] HybridClock initialized: now 1790681235011768 us; error 42 us; skew 500 ppm
I20260929 11:27:15.012204 30531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:15 dist-test-slave-qk05 krb5kdc[30291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681235, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:15.015856 30531 init.cc:377] Logged in from keytab as oryx/127.23.153.2@KRBTEST.COM (short username oryx)
I20260929 11:27:15.017668 30531 webserver.cc:460] Webserver started at http://127.23.153.2:40751/ using document root <none> and password file <none>
I20260929 11:27:15.018468 30531 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:15.018596 30531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:15.018913 30531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:15.023007 30531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "e690037a82ee4b08bcc688edc9073faa"
format_stamp: "Formatted at 2026-09-29 11:27:15 on dist-test-slave-qk05"
server_key: "7ed4f6e5e6e7622fb9b025b3d86735f4"
server_key_iv: "2ddcdbd68b951ea11a63becba7c0f8b3"
server_key_version: "encryptionkey@0"
I20260929 11:27:15.024111 30531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "e690037a82ee4b08bcc688edc9073faa"
format_stamp: "Formatted at 2026-09-29 11:27:15 on dist-test-slave-qk05"
server_key: "7ed4f6e5e6e7622fb9b025b3d86735f4"
server_key_iv: "2ddcdbd68b951ea11a63becba7c0f8b3"
server_key_version: "encryptionkey@0"
I20260929 11:27:15.030431 30531 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 11:27:15.034734 30546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:15.036674 30531 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:27:15.036871 30531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "e690037a82ee4b08bcc688edc9073faa"
format_stamp: "Formatted at 2026-09-29 11:27:15 on dist-test-slave-qk05"
server_key: "7ed4f6e5e6e7622fb9b025b3d86735f4"
server_key_iv: "2ddcdbd68b951ea11a63becba7c0f8b3"
server_key_version: "encryptionkey@0"
I20260929 11:27:15.037073 30531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:15.050225 30531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:15.054932 30531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:15.055292 30531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:15.056358 30531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:15.058032 30531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:15.058122 30531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:15.058235 30531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:15.058283 30531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:15.092792 30531 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:38021
I20260929 11:27:15.092841 30658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:38021 every 8 connection(s)
I20260929 11:27:15.094283 30531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:27:15.094769 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 30531
I20260929 11:27:15.094923 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:27:15.095539 24164 external_mini_cluster.cc:1428] Setting key 54fedccfcccd4805939a0f99f24d1fde
Sep 29 11:27:15 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681235, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.23.153.3@KRBTEST.COM; defaulting to no policy
I20260929 11:27:15.119217 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:15.097413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:37041 (local address 127.23.153.62:41979)
0929 11:27:15.099805 (+ 2392us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:15.099812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:15.099837 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:27:15.101252 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:15.101258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:15.101352 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:15.101447 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:15.103166 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:15.103949 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:15.105896 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:15.106343 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:15.106487 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:15.111669 (+ 5182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:15.111678 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:15.111682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:15.111741 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:15.115188 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:15.116406 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:15.116420 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:15.116427 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:15.116602 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:15.117226 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:15.117230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:15.117232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:15.117538 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:15.117679 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:15.118504 (+ 825us) server_negotiation.cc:299] Negotiation successful
0929 11:27:15.118760 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2258,"thread_start_us":2106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.153.3@KRBTEST.COM" created.
I20260929 11:27:15.120324 30662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:15.098011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.2:37041)
0929 11:27:15.098935 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:15.098985 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:15.100905 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:15.101741 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:15.101751 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:15.102308 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:15.102936 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:15.102956 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:15.104127 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:15.104131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:15.105682 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:15.105702 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:15.106622 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:15.106629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:15.106903 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:15.107725 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:15.107751 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:15.111197 (+ 3446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:15.115408 (+ 4211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:15.115430 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:15.115456 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:15.116206 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:15.116735 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:15.116741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:15.116746 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:15.117080 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:15.117680 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:15.117690 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:15.118094 (+ 404us) client_negotiation.cc:769] Sending connection context
0929 11:27:15.118359 (+ 265us) client_negotiation.cc:240] Negotiation successful
0929 11:27:15.118986 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":383,"threads_started":1}
I20260929 11:27:15.122299 30659 heartbeater.cc:347] Connected to a master server at 127.23.153.62:41979
I20260929 11:27:15.122879 30659 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:15.123941 30659 heartbeater.cc:511] Master 127.23.153.62:41979 requested a full tablet report, sending...
I20260929 11:27:15.125672 30336 ts_manager.cc:194] Registered new tserver with Master: e690037a82ee4b08bcc688edc9073faa (127.23.153.2:38021)
I20260929 11:27:15.126529 30336 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:37041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.3.keytab.
Entry for principal oryx/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.3.keytab.
I20260929 11:27:15.146425 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:41979
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.3.keytab
--principal=oryx/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:15.514086 30665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:15.514417 30665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:15.514539 30665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:15.524461 30665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:15.524611 30665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:15.524739 30665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:27:15.536540 30665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.3.keytab
--principal=oryx/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:41979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:15.538053 30665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:15.540149 30665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:15.556602 30670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:15.556754 30673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:15.556639 30671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:15.559069 30665 server_base.cc:1034] running on GCE node
I20260929 11:27:15.559767 30665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:15.560770 30665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:15.561985 30665 hybrid_clock.cc:648] HybridClock initialized: now 1790681235561937 us; error 43 us; skew 500 ppm
I20260929 11:27:15.562323 30665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:15 dist-test-slave-qk05 krb5kdc[30291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681235, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:15.565713 30665 init.cc:377] Logged in from keytab as oryx/127.23.153.3@KRBTEST.COM (short username oryx)
I20260929 11:27:15.567540 30665 webserver.cc:460] Webserver started at http://127.23.153.3:34655/ using document root <none> and password file <none>
I20260929 11:27:15.568346 30665 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:15.568495 30665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:15.568801 30665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:15.572113 30665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "7ab39c924c1b4797bfac4febda508d47"
format_stamp: "Formatted at 2026-09-29 11:27:15 on dist-test-slave-qk05"
server_key: "2ade45ad72a86b6f4f87bf0b62df5829"
server_key_iv: "79880b2ab5ba56704481c8ceaeee3622"
server_key_version: "encryptionkey@0"
I20260929 11:27:15.573048 30665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "7ab39c924c1b4797bfac4febda508d47"
format_stamp: "Formatted at 2026-09-29 11:27:15 on dist-test-slave-qk05"
server_key: "2ade45ad72a86b6f4f87bf0b62df5829"
server_key_iv: "79880b2ab5ba56704481c8ceaeee3622"
server_key_version: "encryptionkey@0"
I20260929 11:27:15.579525 30665 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 11:27:15.584228 30680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:15.586261 30665 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:27:15.586462 30665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "7ab39c924c1b4797bfac4febda508d47"
format_stamp: "Formatted at 2026-09-29 11:27:15 on dist-test-slave-qk05"
server_key: "2ade45ad72a86b6f4f87bf0b62df5829"
server_key_iv: "79880b2ab5ba56704481c8ceaeee3622"
server_key_version: "encryptionkey@0"
I20260929 11:27:15.586674 30665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:15.588228 30525 heartbeater.cc:503] Master 127.23.153.62:41979 was elected leader, sending a full tablet report...
I20260929 11:27:15.622910 30665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:15.627621 30665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:15.627907 30665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:15.628921 30665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:15.630563 30665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:15.630654 30665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:15.630762 30665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:15.630806 30665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:15.668982 30665 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:40199
I20260929 11:27:15.669203 30792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:40199 every 8 connection(s)
I20260929 11:27:15.670339 30665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:27:15.679904 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 30665
I20260929 11:27:15.680059 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:27:15.680481 24164 external_mini_cluster.cc:1428] Setting key 00f46f875882414565ad952148f57203
Sep 29 11:27:15 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681235, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM
I20260929 11:27:15.694777 30795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:15.673475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:57725 (local address 127.23.153.62:41979)
0929 11:27:15.673985 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:15.673991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:15.675143 (+ 1152us) server_negotiation.cc:407] Connection header received
0929 11:27:15.677145 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:15.677150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:15.677235 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:15.677347 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:15.679182 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:15.680074 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:15.681873 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:15.682298 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:15.682661 (+ 363us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:15.687491 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:15.687498 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:15.687501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:15.687547 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:15.690491 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:15.691568 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:15.691573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:15.691576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:15.691718 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:15.692478 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:15.692485 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:15.692488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:15.692997 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:15.693147 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:15.694079 (+ 932us) server_negotiation.cc:299] Negotiation successful
0929 11:27:15.694357 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":231,"threads_started":1}
I20260929 11:27:15.695919 30796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:15.674099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.3:57725)
0929 11:27:15.674928 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:15.674970 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:15.676773 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:15.677604 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:15.677614 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:15.678196 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:15.678874 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:15.678896 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:15.680266 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:15.680272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:15.681693 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:15.681719 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:15.682841 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:15.682853 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:15.683272 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:15.684322 (+ 1050us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:15.684372 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:15.687138 (+ 2766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:15.690727 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:15.690737 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:15.690755 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:15.691421 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:15.691862 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:15.691867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:15.691870 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:15.692153 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:15.693163 (+ 1010us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:15.693170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:15.693600 (+ 430us) client_negotiation.cc:769] Sending connection context
0929 11:27:15.693927 (+ 327us) client_negotiation.cc:240] Negotiation successful
0929 11:27:15.694492 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":370,"threads_started":1}
I20260929 11:27:15.698011 30793 heartbeater.cc:347] Connected to a master server at 127.23.153.62:41979
I20260929 11:27:15.698328 30793 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:15.699182 30793 heartbeater.cc:511] Master 127.23.153.62:41979 requested a full tablet report, sending...
I20260929 11:27:15.700840 30336 ts_manager.cc:194] Registered new tserver with Master: 7ab39c924c1b4797bfac4febda508d47 (127.23.153.3:40199)
I20260929 11:27:15.701814 30336 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:57725
I20260929 11:27:15.710098 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.153.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.153.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.153.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.61.keytab.
Entry for principal oryx/127.23.153.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.61.keytab.
I20260929 11:27:15.763430 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.61:40211
--webserver_interface=127.23.153.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.61.keytab
--principal=oryx/127.23.153.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:41979,127.23.153.61:40211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:27:16.129437 30659 heartbeater.cc:503] Master 127.23.153.62:41979 was elected leader, sending a full tablet report...
W20260929 11:27:16.130425 30799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:16.130867 30799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:16.131049 30799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:16.139051 30799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:27:16.139185 30799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:16.139232 30799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:16.139289 30799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:27:16.139331 30799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:27:16.150627 30799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:41979,127.23.153.61:40211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.61.keytab
--principal=oryx/127.23.153.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.61:40211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.153.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
W20260929 11:27:16.151106 30799 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.153.62:41979,127.23.153.61:40211'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 11:27:16.152293 30799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:16.153898 30799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:16.166780 30804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:16.166841 30805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:16.166808 30807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:16.168021 30799 server_base.cc:1034] running on GCE node
I20260929 11:27:16.168931 30799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:16.170624 30799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:16.171906 30799 hybrid_clock.cc:648] HybridClock initialized: now 1790681236171831 us; error 56 us; skew 500 ppm
I20260929 11:27:16.172443 30799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:16 dist-test-slave-qk05 krb5kdc[30291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681236, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:16.175913 30799 init.cc:377] Logged in from keytab as oryx/127.23.153.61@KRBTEST.COM (short username oryx)
I20260929 11:27:16.177619 30799 webserver.cc:460] Webserver started at http://127.23.153.61:34975/ using document root <none> and password file <none>
I20260929 11:27:16.178398 30799 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:16.178516 30799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:16.178843 30799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:16.182202 30799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/data/instance:
uuid: "b8f3ffad32964656969d4daec256606d"
format_stamp: "Formatted at 2026-09-29 11:27:16 on dist-test-slave-qk05"
server_key: "59580f86ed7fc94b7b546e21a8290752"
server_key_iv: "8262ab42565fa72be4bcaf70d47f6133"
server_key_version: "encryptionkey@0"
I20260929 11:27:16.183173 30799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/wal/instance:
uuid: "b8f3ffad32964656969d4daec256606d"
format_stamp: "Formatted at 2026-09-29 11:27:16 on dist-test-slave-qk05"
server_key: "59580f86ed7fc94b7b546e21a8290752"
server_key_iv: "8262ab42565fa72be4bcaf70d47f6133"
server_key_version: "encryptionkey@0"
I20260929 11:27:16.189949 30799 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260929 11:27:16.194012 30814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:16.196064 30799 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:27:16.196275 30799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/wal
uuid: "b8f3ffad32964656969d4daec256606d"
format_stamp: "Formatted at 2026-09-29 11:27:16 on dist-test-slave-qk05"
server_key: "59580f86ed7fc94b7b546e21a8290752"
server_key_iv: "8262ab42565fa72be4bcaf70d47f6133"
server_key_version: "encryptionkey@0"
I20260929 11:27:16.196529 30799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:16.215806 30799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:16.220396 30799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:16.220670 30799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:16.240047 30799 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.61:40211
I20260929 11:27:16.240060 30865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.61:40211 every 8 connection(s)
I20260929 11:27:16.241667 30799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/data/info.pb
I20260929 11:27:16.245838 30866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:16.246997 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 30799
I20260929 11:27:16.256083 30866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:27:16 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.153.61@KRBTEST.COM
Sep 29 11:27:16 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681236, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.61@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM
I20260929 11:27:16.273706 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:16.250052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.23.153.61:40211)
0929 11:27:16.250939 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:16.250953 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:16.251007 (+ 54us) server_negotiation.cc:407] Connection header received
0929 11:27:16.252701 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:16.252730 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:16.253222 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:16.253612 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:16.255607 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.257038 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.258874 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.260102 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.260319 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:16.265557 (+ 5238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:16.265646 (+ 89us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:16.265687 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:16.265756 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:16.269183 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:16.270080 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:16.270088 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:16.270096 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:16.270277 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:16.270685 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:16.270690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:16.270693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:16.271341 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:16.271638 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:16.272134 (+ 496us) server_negotiation.cc:299] Negotiation successful
0929 11:27:16.272616 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":385,"threads_started":1}
I20260929 11:27:16.275363 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:16.259398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50164 (local address 127.23.153.62:41979)
0929 11:27:16.259856 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:16.259862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:16.260573 (+ 711us) server_negotiation.cc:407] Connection header received
0929 11:27:16.260764 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:16.260770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:16.260852 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:16.260964 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:16.262338 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.263172 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.264940 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.265410 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.265540 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:16.269460 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:16.269467 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:16.269470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:16.269514 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:16.272108 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:16.273019 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:16.273025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:16.273028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:16.273152 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:16.273652 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:16.273656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:16.273659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:16.274066 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:16.274212 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:16.274696 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 11:27:16.274963 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":218,"threads_started":1}
I20260929 11:27:16.275411 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:16.259448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.0.0.1:50164)
0929 11:27:16.260248 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:16.260415 (+ 167us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:16.260602 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:16.261073 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:16.261082 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:16.261880 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:16.262139 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:16.262149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.263337 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.263341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:16.264745 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:16.264774 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.265658 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.265664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:16.265881 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:16.266599 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:16.266618 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:16.269210 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:16.272252 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:16.272265 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:16.272273 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:16.272897 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:16.273265 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:16.273271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:16.273275 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:16.273514 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:16.274205 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:16.274214 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:16.274400 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 11:27:16.274579 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 11:27:16.274955 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":398,"threads_started":1}
I20260929 11:27:16.277087 30866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:27:16 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681236, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.61@KRBTEST.COM for oryx/127.23.153.61@KRBTEST.COM
I20260929 11:27:16.290802 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:16.277816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:40211 (local address 127.0.0.1:60180)
0929 11:27:16.278007 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:16.278031 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:16.278148 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:16.278443 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:16.278447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:16.278843 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:16.278975 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:16.278984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.279989 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.279993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:16.280885 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:16.280898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.281630 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.281635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:16.281811 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:16.282468 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:16.282486 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:16.284741 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:16.288610 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:16.288615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:16.288618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:16.289054 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:16.289408 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:16.289411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:16.289413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:16.289538 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:16.290042 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:16.290045 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:16.290178 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 11:27:16.290288 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 11:27:16.290538 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 11:27:16.291179 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:16.277981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.23.153.61:40211)
0929 11:27:16.278124 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:16.278129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:16.278147 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:27:16.278256 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:16.278261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:16.278312 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:16.278407 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:16.279117 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.279830 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.281035 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.281473 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.281590 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:16.285122 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:16.285129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:16.285134 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:16.285186 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:16.288407 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:16.289187 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:16.289191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:16.289193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:16.289316 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:16.289630 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:16.289633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:16.289635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:16.289946 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:16.290054 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:16.290371 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 11:27:16.290798 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":55}
I20260929 11:27:16.296626 30866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: Bootstrap starting.
I20260929 11:27:16.300048 30866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:16.301218 30866 log.cc:824] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:16.304988 30866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: No bootstrap required, opened a new log
I20260929 11:27:16.312476 30866 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } }
I20260929 11:27:16.312832 30866 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:16.312908 30866 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f3ffad32964656969d4daec256606d, State: Initialized, Role: FOLLOWER
I20260929 11:27:16.313472 30866 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } }
I20260929 11:27:16.314338 30873 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } } }
I20260929 11:27:16.314649 30873 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:27:16.315438 30866 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: configured and running, proceeding with master startup.
W20260929 11:27:16.325526 30884 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:27:16.325696 30884 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:27:16.328364 30799 master_runner.cc:428] Detected that this master b8f3ffad32964656969d4daec256606d is joining an existing cluster
I20260929 11:27:16.328495 30799 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.153.61:40211
I20260929 11:27:16.350190 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:16.333488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50170 (local address 127.23.153.62:41979)
0929 11:27:16.333848 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:16.333855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:16.334279 (+ 424us) server_negotiation.cc:407] Connection header received
0929 11:27:16.334532 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:16.334538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:16.334623 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:16.334688 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:16.336202 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.337272 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.339362 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.339980 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.340081 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:16.342977 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:16.342984 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:16.342989 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:16.343053 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:16.346453 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:16.347654 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:16.347661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:16.347665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:16.347857 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:16.348371 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:16.348377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:16.348380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:16.348767 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:16.348927 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:16.349575 (+ 648us) server_negotiation.cc:299] Negotiation successful
0929 11:27:16.349861 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260929 11:27:16.351542 30890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:16.333423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.0.0.1:50170)
0929 11:27:16.334160 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:16.334180 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:16.334393 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:16.334904 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:16.334909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:16.335620 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:16.336019 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:16.336032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.337504 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.337510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:16.339165 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:16.339182 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.340453 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.340459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:16.340623 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:16.341494 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:16.341517 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:16.342718 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:16.346645 (+ 3927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:16.346650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:16.346655 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:16.347481 (+ 826us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:16.348022 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:16.348027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:16.348031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:16.348219 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:16.349023 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:16.349029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:16.349242 (+ 213us) client_negotiation.cc:769] Sending connection context
0929 11:27:16.350471 (+ 1229us) client_negotiation.cc:240] Negotiation successful
0929 11:27:16.350981 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":86,"thread_start_us":430,"threads_started":1}
WARNING: no policy specified for oryx/127.23.153.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.153.60@KRBTEST.COM" created.
I20260929 11:27:16.366807 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:16.334751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.23.153.61:40211)
0929 11:27:16.334930 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:16.334936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:16.346152 (+ 11216us) server_negotiation.cc:407] Connection header received
0929 11:27:16.346646 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:16.346651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:16.346722 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:16.346791 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:16.348541 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.349656 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.352360 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.352937 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.353066 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:16.355319 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:16.355327 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:16.355332 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:16.355379 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:16.359257 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:16.364222 (+ 4965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:16.364229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:16.364233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:16.364454 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:16.364912 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:16.364917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:16.364920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:16.365336 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:16.365504 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:16.366148 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 11:27:16.366436 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:27:16.373677 30891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:16.334414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:40211 (local address 127.0.0.1:60186)
0929 11:27:16.346038 (+ 11624us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:16.346058 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:16.346501 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:16.347008 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:16.347033 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:16.347985 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:16.348372 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:16.348385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.349880 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.349886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:16.352163 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:16.352183 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.353136 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.353142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:16.353332 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:16.353922 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:16.353939 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:16.354966 (+ 1027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:16.363181 (+ 8215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:16.363188 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:16.363193 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:16.364024 (+ 831us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:16.364610 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:16.364615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:16.364619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:16.364795 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:16.365572 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:16.365576 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:16.365818 (+ 242us) client_negotiation.cc:769] Sending connection context
0929 11:27:16.372674 (+ 6856us) client_negotiation.cc:240] Negotiation successful
0929 11:27:16.373079 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11455,"mutex_wait_us":26,"thread_start_us":11744,"threads_started":1}
Sep 29 11:27:16 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681233, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.62@KRBTEST.COM for oryx/127.23.153.61@KRBTEST.COM
I20260929 11:27:16.384546 30893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:16.359444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:40211 (local address 127.0.0.1:60202)
0929 11:27:16.360122 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:16.360167 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:16.360380 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:16.367759 (+ 7379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:16.367771 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:16.368739 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:16.369076 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:16.369091 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.370739 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.370745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:16.372463 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:16.372486 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.373728 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.373735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:16.373964 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:16.375091 (+ 1127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:16.375117 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:16.378284 (+ 3167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:16.381566 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:16.381584 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:16.381592 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:16.382045 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:16.382438 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:16.382443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:16.382447 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:16.382600 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:16.383179 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:16.383188 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:16.383497 (+ 309us) client_negotiation.cc:769] Sending connection context
0929 11:27:16.383700 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 11:27:16.384098 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":729,"threads_started":1}
I20260929 11:27:16.384510 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:16.359834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60202 (local address 127.23.153.61:40211)
0929 11:27:16.367104 (+ 7270us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:16.367110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:16.367130 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:27:16.367241 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:16.367247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:16.367342 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:16.367448 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:16.369323 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.370514 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.372666 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.373372 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.373521 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:16.378557 (+ 5036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:16.378566 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:16.378571 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:16.378619 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:16.381383 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:16.382176 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:16.382182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:16.382186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:16.382323 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:16.382706 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:16.382710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:16.382712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:16.383009 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:16.383187 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:16.383882 (+ 695us) server_negotiation.cc:299] Negotiation successful
0929 11:27:16.384117 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7134,"thread_start_us":11933,"threads_started":1}
I20260929 11:27:16.387089 30336 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.153.61:40211: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b8f3ffad32964656969d4daec256606d"
member_type: NON_VOTER
last_known_addr {
host: "127.23.153.61"
port: 40211
}
attrs {
promote: true
}
}
dest_uuid: "bd60a4cf233742daa64ac4c57ea49ea7"
cas_config_opid_index: -1
I20260929 11:27:16.388221 30336 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.153.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.60.keytab.
Entry for principal oryx/127.23.153.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.60.keytab.
I20260929 11:27:16.391613 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.60:38757
--webserver_interface=127.23.153.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.60.keytab
--principal=oryx/127.23.153.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:41979,127.23.153.61:40211,127.23.153.60:38757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:27:16.399667 30897 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b8f3ffad32964656969d4daec256606d (127.23.153.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true } } }
I20260929 11:27:16.401880 30841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:16.405558 30841 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 1 FOLLOWER]: Refusing update from remote peer bd60a4cf233742daa64ac4c57ea49ea7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 11:27:16.406641 30898 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:16.409032 30897 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.153.61:40211
I20260929 11:27:16.409538 30897 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true } } }
I20260929 11:27:16.409773 30897 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:16.409888 30799 master.cc:537] Master@127.23.153.61:40211 shutting down...
I20260929 11:27:16.432013 30841 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b8f3ffad32964656969d4daec256606d (127.23.153.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true } } }
I20260929 11:27:16.452598 30873 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd60a4cf233742daa64ac4c57ea49ea7. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true } } }
I20260929 11:27:16.452971 30873 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: This master's current role is: LEARNER
I20260929 11:27:16.452577 30903 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true } } }
I20260929 11:27:16.453264 30903 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: This master's current role is: LEARNER
I20260929 11:27:16.453506 30907 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true } } }
I20260929 11:27:16.453864 30907 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: This master's current role is: LEARNER
I20260929 11:27:16.456442 30799 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 1 LEARNER]: Raft consensus shutting down.
I20260929 11:27:16.457039 30799 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 1 LEARNER]: Raft consensus is shut down!
I20260929 11:27:16.457226 30799 tablet_replica.cc:330] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: stopping tablet replica
I20260929 11:27:16.467821 30799 master.cc:552] Master@127.23.153.61:40211 shutdown complete.
I20260929 11:27:16.467940 30799 master_runner.cc:305] Clearing existing system tablet
I20260929 11:27:16.468364 30799 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:16.468466 30799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:16.471853 30799 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 11:27:16.473730 30909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:16.474146 30799 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:16.474505 30799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260929 11:27:16.474627 30799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/wal
uuid: "b8f3ffad32964656969d4daec256606d"
format_stamp: "Formatted at 2026-09-29 11:27:16 on dist-test-slave-qk05"
server_key: "59580f86ed7fc94b7b546e21a8290752"
server_key_iv: "8262ab42565fa72be4bcaf70d47f6133"
server_key_version: "encryptionkey@0"
I20260929 11:27:16.480672 30799 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:16.488541 30799 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 11:27:16.488920 30799 log.cc:1196] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 11:27:16.489497 30799 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: Deleting consensus metadata
I20260929 11:27:16.489812 30799 master_runner.cc:315] Copying system tablet from 127.23.153.62:41979
I20260929 11:27:16.492924 30799 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: tablet copy: Beginning tablet copy session from remote peer at address 127.23.153.62:41979
I20260929 11:27:16.519210 30915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:16.494274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.0.0.1:50180)
0929 11:27:16.494973 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:16.494993 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:16.495251 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:16.495697 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:16.495702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:16.496296 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:16.496622 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:16.496635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.498203 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.498209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:16.499807 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:16.499832 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.500935 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.500941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:16.501137 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:16.508802 (+ 7665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:16.508848 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:16.510565 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:16.515740 (+ 5175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:16.515747 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:16.515751 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:16.516406 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:16.516965 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:16.516974 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:16.516977 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:16.517153 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:16.517849 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:16.517854 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:16.518062 (+ 208us) client_negotiation.cc:769] Sending connection context
0929 11:27:16.518259 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 11:27:16.518695 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":383,"threads_started":1}
I20260929 11:27:16.519537 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:16.494325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50180 (local address 127.23.153.62:41979)
0929 11:27:16.494520 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:16.494526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:16.495144 (+ 618us) server_negotiation.cc:407] Connection header received
0929 11:27:16.495436 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:16.495443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:16.495547 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:16.495654 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:16.496838 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.498011 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.500026 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:16.500738 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:16.500890 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:16.510941 (+ 10051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:16.510950 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:16.510955 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:16.511011 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:16.515489 (+ 4478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:16.516592 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:16.516598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:16.516602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:16.516788 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:16.517289 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:16.517294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:16.517298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:16.517703 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:16.517859 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:16.518414 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 11:27:16.518676 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 11:27:16.520799 30356 tablet_copy_service.cc:140] P bd60a4cf233742daa64ac4c57ea49ea7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b8f3ffad32964656969d4daec256606d ({username='oryx', principal='oryx/127.23.153.61@KRBTEST.COM'} at 127.0.0.1:50180)
I20260929 11:27:16.521135 30356 tablet_copy_service.cc:161] P bd60a4cf233742daa64ac4c57ea49ea7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b8f3ffad32964656969d4daec256606d at {username='oryx', principal='oryx/127.23.153.61@KRBTEST.COM'} at 127.0.0.1:50180: session id = b8f3ffad32964656969d4daec256606d-00000000000000000000000000000000
I20260929 11:27:16.526461 30356 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 11:27:16.530056 30799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 11:27:16.538975 30799 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: tablet copy: Starting download of 0 data blocks...
I20260929 11:27:16.539367 30799 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: tablet copy: Starting download of 1 WAL segments...
I20260929 11:27:16.542927 30799 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 11:27:16.548990 30356 tablet_copy_service.cc:342] P bd60a4cf233742daa64ac4c57ea49ea7: Request end of tablet copy session b8f3ffad32964656969d4daec256606d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.153.61@KRBTEST.COM'} at 127.0.0.1:50180
I20260929 11:27:16.549575 30356 tablet_copy_service.cc:433] P bd60a4cf233742daa64ac4c57ea49ea7: ending tablet copy session b8f3ffad32964656969d4daec256606d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b8f3ffad32964656969d4daec256606d
I20260929 11:27:16.564990 30799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:16.569502 30916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:16.570358 30917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:16.579299 30799 server_base.cc:1034] running on GCE node
W20260929 11:27:16.580430 30920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:16.580961 30799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:16.581753 30799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:16.583117 30799 hybrid_clock.cc:648] HybridClock initialized: now 1790681236582993 us; error 37 us; skew 500 ppm
I20260929 11:27:16.583506 30799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:16 dist-test-slave-qk05 krb5kdc[30291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681236, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:16.585824 30799 init.cc:377] Logged in from keytab as oryx/127.23.153.61@KRBTEST.COM (short username oryx)
I20260929 11:27:16.595468 30799 webserver.cc:460] Webserver started at http://127.23.153.61:44013/ using document root <none> and password file <none>
I20260929 11:27:16.595911 30799 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:16.596000 30799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:16.599310 30799 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 11:27:16.601467 30926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:16.601943 30799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260929 11:27:16.602126 30799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/wal
uuid: "b8f3ffad32964656969d4daec256606d"
format_stamp: "Formatted at 2026-09-29 11:27:16 on dist-test-slave-qk05"
server_key: "59580f86ed7fc94b7b546e21a8290752"
server_key_iv: "8262ab42565fa72be4bcaf70d47f6133"
server_key_version: "encryptionkey@0"
I20260929 11:27:16.602394 30799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:16.617919 30799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:16.623202 30799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:16.659900 30799 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.61:40211
I20260929 11:27:16.659946 30977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.61:40211 every 8 connection(s)
I20260929 11:27:16.661173 30799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-1/data/info.pb
I20260929 11:27:16.666807 30978 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:27:16.674811 30978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: Bootstrap starting.
I20260929 11:27:16.691097 30978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:27:16.692090 30978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: Bootstrap complete.
I20260929 11:27:16.693730 30978 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true } }
I20260929 11:27:16.694044 30978 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b8f3ffad32964656969d4daec256606d, State: Initialized, Role: LEARNER
I20260929 11:27:16.694399 30978 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true } }
I20260929 11:27:16.695358 30981 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true } } }
I20260929 11:27:16.695667 30978 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:27:16.695863 30981 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: This master's current role is: LEARNER
I20260929 11:27:16.704022 30992 catalog_manager.cc:1259] Loaded cluster ID: fb7a3e2cc82d4b3ba2e460537a75d64b
I20260929 11:27:16.704537 30992 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: loading cluster ID for follower catalog manager: success
I20260929 11:27:16.705570 30793 heartbeater.cc:503] Master 127.23.153.62:41979 was elected leader, sending a full tablet report...
I20260929 11:27:16.709280 30992 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: acquiring CA information for follower catalog manager: success
I20260929 11:27:16.712654 30992 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 11:27:16.852339 30899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:16.852666 30899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:16.852784 30899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:16.860785 30899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:27:16.860929 30899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:16.860980 30899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:16.861037 30899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:27:16.861080 30899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:27:16.872040 30899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:41979,127.23.153.61:40211,127.23.153.60:38757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.60.keytab
--principal=oryx/127.23.153.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.60:38757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.153.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:16.873453 30899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:16.874778 30899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:16.889779 30998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:16.890103 31000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:16.890396 30997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:16.891106 30899 server_base.cc:1034] running on GCE node
I20260929 11:27:16.892060 30899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:16.893791 30899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:16.895094 30899 hybrid_clock.cc:648] HybridClock initialized: now 1790681236894985 us; error 66 us; skew 500 ppm
I20260929 11:27:16.895522 30899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:16 dist-test-slave-qk05 krb5kdc[30291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681236, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:16.899050 30899 init.cc:377] Logged in from keytab as oryx/127.23.153.60@KRBTEST.COM (short username oryx)
I20260929 11:27:16.900974 30899 webserver.cc:460] Webserver started at http://127.23.153.60:41487/ using document root <none> and password file <none>
I20260929 11:27:16.901937 30899 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:16.902101 30899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:16.902511 30899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:16.907399 30899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/data/instance:
uuid: "8c00c49401624509b0e2c682d3c361fb"
format_stamp: "Formatted at 2026-09-29 11:27:16 on dist-test-slave-qk05"
server_key: "a708594878309b4daa801ed291680ce8"
server_key_iv: "6ff453a2c9c4ba52e6f38c9fa1e589eb"
server_key_version: "encryptionkey@0"
I20260929 11:27:16.908573 30899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/wal/instance:
uuid: "8c00c49401624509b0e2c682d3c361fb"
format_stamp: "Formatted at 2026-09-29 11:27:16 on dist-test-slave-qk05"
server_key: "a708594878309b4daa801ed291680ce8"
server_key_iv: "6ff453a2c9c4ba52e6f38c9fa1e589eb"
server_key_version: "encryptionkey@0"
I20260929 11:27:16.916142 30899 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260929 11:27:16.920361 31007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:16.922184 30899 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 11:27:16.922415 30899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/wal
uuid: "8c00c49401624509b0e2c682d3c361fb"
format_stamp: "Formatted at 2026-09-29 11:27:16 on dist-test-slave-qk05"
server_key: "a708594878309b4daa801ed291680ce8"
server_key_iv: "6ff453a2c9c4ba52e6f38c9fa1e589eb"
server_key_version: "encryptionkey@0"
I20260929 11:27:16.922643 30899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:16.965973 30899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:16.972004 30899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:16.972431 30899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:16.992110 30899 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.60:38757
I20260929 11:27:16.992126 31058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.60:38757 every 8 connection(s)
I20260929 11:27:16.993693 30899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/data/info.pb
I20260929 11:27:16.998296 31059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:16.998481 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 30899
I20260929 11:27:17.011051 31059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:27:17.014573 31065 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true } } }
I20260929 11:27:17.014869 31065 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: This master's current role is: LEADER
Sep 29 11:27:17 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.153.60@KRBTEST.COM
I20260929 11:27:17.030772 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.000799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36378 (local address 127.23.153.60:38757)
0929 11:27:17.001676 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:17.001689 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:17.001738 (+ 49us) server_negotiation.cc:407] Connection header received
0929 11:27:17.004164 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:17.004198 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:17.004874 (+ 676us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:17.005385 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:17.007326 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.009432 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.011543 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.012907 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.013095 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.019857 (+ 6762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:17.019871 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:17.019896 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:17.019963 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:17.024403 (+ 4440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.025524 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.025535 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.025548 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.025832 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.026340 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.026345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.026349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.027229 (+ 880us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:17.027574 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:17.028419 (+ 845us) server_negotiation.cc:299] Negotiation successful
0929 11:27:17.029006 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":391,"threads_started":1}
I20260929 11:27:17.038071 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.008743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.23.153.61:40211)
0929 11:27:17.030035 (+ 21292us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:17.030042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:17.030593 (+ 551us) server_negotiation.cc:407] Connection header received
0929 11:27:17.030786 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:17.030792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:17.030886 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:17.031002 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:17.032838 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.034202 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.036077 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.036999 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.037160 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.037420 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:17.037607 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 11:27:17.037754 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21144,"thread_start_us":21000,"threads_started":1}
I20260929 11:27:17.038091 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.008336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:40211 (local address 127.0.0.1:60208)
0929 11:27:17.030502 (+ 22166us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:17.030521 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:17.030659 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:17.031575 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:17.031580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:17.031624 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:17.032631 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.032644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.034377 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.034381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.035900 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.035915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.037191 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.037197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.037399 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.037461 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 11:27:17.037547 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 11:27:17.037701 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22051,"mutex_wait_us":1,"thread_start_us":21895,"threads_started":1}
Sep 29 11:27:17 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681236, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.60@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM
I20260929 11:27:17.039163 30981 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd60a4cf233742daa64ac4c57ea49ea7. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true } } }
I20260929 11:27:17.039412 30981 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: This master's current role is: LEARNER
I20260929 11:27:17.040453 31065 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7: attempting to promote NON_VOTER b8f3ffad32964656969d4daec256606d to VOTER
I20260929 11:27:17.041507 31065 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } }
I20260929 11:27:17.043751 30953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 1 LEARNER]: Refusing update from remote peer bd60a4cf233742daa64ac4c57ea49ea7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 11:27:17.044418 31063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 11:27:17.048071 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.015388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50188 (local address 127.23.153.62:41979)
0929 11:27:17.015593 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:17.015600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:17.026814 (+ 11214us) server_negotiation.cc:407] Connection header received
0929 11:27:17.028093 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:17.028100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:17.028199 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:17.028276 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:17.032295 (+ 4019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.033574 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.035405 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.035869 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.035982 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.039974 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:17.039982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:17.039987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:17.040034 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:17.043967 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.045265 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.045274 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.045277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.045687 (+ 410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.046144 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.046151 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.046154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.046551 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:17.046723 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:17.047494 (+ 771us) server_negotiation.cc:299] Negotiation successful
0929 11:27:17.047729 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:27:17.048386 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.015976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.0.0.1:50188)
0929 11:27:17.026619 (+ 10643us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:17.026687 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:17.027888 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:17.028558 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:17.028566 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:17.031873 (+ 3307us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:17.032132 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.032143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.033985 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.033990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.035244 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.035263 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.036061 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.036067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.036256 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.036910 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:17.036929 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:17.039506 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:17.044263 (+ 4757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.044278 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.044286 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.044967 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.045831 (+ 864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.045835 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.045838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.046000 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.046736 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:17.046746 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:17.046978 (+ 232us) client_negotiation.cc:769] Sending connection context
0929 11:27:17.047474 (+ 496us) client_negotiation.cc:240] Negotiation successful
0929 11:27:17.047875 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10263,"thread_start_us":10863,"threads_started":1}
I20260929 11:27:17.048760 31065 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b8f3ffad32964656969d4daec256606d (127.23.153.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } }
I20260929 11:27:17.049762 30953 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b8f3ffad32964656969d4daec256606d (127.23.153.61) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } }
I20260929 11:27:17.050195 31059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:27:17.052345 31065 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } }
I20260929 11:27:17.052333 31063 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } }
I20260929 11:27:17.052619 31065 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:17.052646 31063 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:17.054024 30981 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } }
I20260929 11:27:17.054275 30981 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: This master's current role is: FOLLOWER
Sep 29 11:27:17 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681236, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.60@KRBTEST.COM for oryx/127.23.153.61@KRBTEST.COM
I20260929 11:27:17.064735 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.050911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:40211 (local address 127.0.0.1:60224)
0929 11:27:17.051538 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:17.051558 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:17.051693 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:17.052085 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:17.052091 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:17.052594 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:17.052932 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.052947 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.054207 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.054213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.055289 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.055307 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.056039 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.056045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.056245 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.056829 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:17.056846 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:17.058968 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:17.062152 (+ 3184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.062158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.062162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.062577 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.063037 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.063042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.063046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.063193 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.063746 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:17.063750 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:17.063917 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 11:27:17.064036 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 11:27:17.064389 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475}
I20260929 11:27:17.064757 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.050992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.23.153.61:40211)
0929 11:27:17.051260 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:17.051266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:17.051653 (+ 387us) server_negotiation.cc:407] Connection header received
0929 11:27:17.051843 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:17.051849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:17.051935 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:17.052046 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:17.053095 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.054030 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.055455 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.055867 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.055985 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.059266 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:17.059272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:17.059275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:17.059317 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:17.061957 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.062731 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.062735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.062737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.062887 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.063320 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.063323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.063325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.063621 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:17.063771 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:17.064162 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 11:27:17.064395 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260929 11:27:17.065919 31059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:27:17 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681236, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.60@KRBTEST.COM for oryx/127.23.153.60@KRBTEST.COM
I20260929 11:27:17.079571 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.066740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:38757 (local address 127.0.0.1:36386)
0929 11:27:17.066912 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:17.066931 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:17.067097 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:17.067614 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:17.067619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:17.068114 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:17.068269 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.068279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.069589 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.069594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.070666 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.070694 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.071604 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.071610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.071822 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.072392 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:17.072408 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:17.074409 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:17.077275 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.077280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.077284 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.077634 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.077966 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.077971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.077975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.078123 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.078608 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:17.078612 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:17.078745 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 11:27:17.078857 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 11:27:17.079153 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 11:27:17.079625 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.066715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36386 (local address 127.23.153.60:38757)
0929 11:27:17.066911 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:17.066917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:17.067056 (+ 139us) server_negotiation.cc:407] Connection header received
0929 11:27:17.067214 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:17.067219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:17.067417 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:17.067554 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:17.068411 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.069448 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.070909 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.071424 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.071554 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.074601 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:17.074605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:17.074609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:17.074646 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:17.077114 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.077745 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.077751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.077754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.077870 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.078217 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.078221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.078225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.078502 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:17.078639 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:17.078941 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 11:27:17.079317 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":156,"server-negotiator.queue_time_us":58}
I20260929 11:27:17.084816 31059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: Bootstrap starting.
I20260929 11:27:17.088266 31059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:17.089344 31059 log.cc:824] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:17.092525 31059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: No bootstrap required, opened a new log
I20260929 11:27:17.100544 31059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } }
I20260929 11:27:17.100960 31059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:17.101087 31059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c00c49401624509b0e2c682d3c361fb, State: Initialized, Role: FOLLOWER
I20260929 11:27:17.101697 31059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } }
I20260929 11:27:17.102658 31072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } } }
I20260929 11:27:17.103053 31072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:27:17.104195 31059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: configured and running, proceeding with master startup.
W20260929 11:27:17.114555 31083 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:27:17.114702 31083 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:27:17.119915 30899 master_runner.cc:428] Detected that this master 8c00c49401624509b0e2c682d3c361fb is joining an existing cluster
I20260929 11:27:17.120074 30899 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.153.60:38757
I20260929 11:27:17.137948 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 30397
I20260929 11:27:17.145823 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.125437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50190 (local address 127.23.153.62:41979)
0929 11:27:17.125626 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:17.125633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:17.125662 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:27:17.125944 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:17.125951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:17.126035 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:17.126108 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:17.127677 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.128876 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.131219 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.131950 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.132065 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.136974 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:17.136982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:17.136987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:17.137029 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:17.141673 (+ 4644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.142840 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.142847 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.142852 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.143079 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.143632 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.143639 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.143643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.144073 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:17.144244 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:17.145173 (+ 929us) server_negotiation.cc:299] Negotiation successful
0929 11:27:17.145440 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 11:27:17.147048 31089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.124870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.0.0.1:50190)
0929 11:27:17.125573 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:17.125599 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:17.125751 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:17.126332 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:17.126336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:17.127094 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:17.127463 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.127477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.129165 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.129171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.130986 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.131006 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.134323 (+ 3317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.134330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.134528 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.135333 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:17.135354 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:17.136722 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:17.141958 (+ 5236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.141964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.141969 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.142646 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.143257 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.143263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.143266 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.143497 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.144427 (+ 930us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:17.144433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:17.144723 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 11:27:17.146115 (+ 1392us) client_negotiation.cc:240] Negotiation successful
0929 11:27:17.146513 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":787,"threads_started":1}
I20260929 11:27:17.161847 31062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.124870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36388 (local address 127.23.153.60:38757)
0929 11:27:17.125084 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:17.125092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:17.139871 (+ 14779us) server_negotiation.cc:407] Connection header received
0929 11:27:17.140274 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:17.140280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:17.140356 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:17.140422 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:17.142093 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.143452 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.145631 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.146461 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.146573 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.154441 (+ 7868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:17.154456 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:17.154463 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:17.154508 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:17.158692 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.159602 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.159608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.159611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.159775 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.160173 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.160178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.160182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.160561 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:17.160729 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:17.161228 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 11:27:17.161466 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:27:17.163484 31091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.124903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:38757 (local address 127.0.0.1:36388)
0929 11:27:17.139757 (+ 14854us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:17.139782 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:17.140133 (+ 351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:17.140635 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:17.140641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:17.141475 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:17.141929 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.141944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.143699 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.143705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.145453 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.145470 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.147509 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.147516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.147730 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.148828 (+ 1098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:17.148851 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:17.150178 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:17.158852 (+ 8674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.158857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.158862 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.159448 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.159911 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.159915 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.159919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.160057 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.160808 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:17.160812 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:17.160978 (+ 166us) client_negotiation.cc:769] Sending connection context
0929 11:27:17.162295 (+ 1317us) client_negotiation.cc:240] Negotiation successful
0929 11:27:17.162973 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14663,"spinlock_wait_cycles":41344,"thread_start_us":23307,"threads_started":1}
Sep 29 11:27:17 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681233, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.62@KRBTEST.COM for oryx/127.23.153.60@KRBTEST.COM
Sep 29 11:27:17 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.153.1@KRBTEST.COM
I20260929 11:27:17.172673 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.126589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60234 (local address 127.23.153.61:40211)
0929 11:27:17.126770 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:17.126777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:17.148644 (+ 21867us) server_negotiation.cc:407] Connection header received
0929 11:27:17.150572 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:17.150592 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:17.150686 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:17.150760 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:17.152688 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.154014 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.156433 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.157073 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.157180 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.165537 (+ 8357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:17.165545 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:17.165549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:17.165598 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:17.169185 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.170079 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.170085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.170088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.170257 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.170776 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.170782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.170785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.171197 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:17.171344 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:17.172052 (+ 708us) server_negotiation.cc:299] Negotiation successful
0929 11:27:17.172309 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:27:17.173743 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.126214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:40211 (local address 127.0.0.1:60234)
0929 11:27:17.148518 (+ 22304us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:17.148537 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:17.150391 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:17.150968 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:17.150973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:17.151768 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:17.152459 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.152477 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.154378 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.154384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.156214 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.156229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.157479 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.157485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.157690 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.158312 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:17.158335 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:17.165283 (+ 6948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:17.169352 (+ 4069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.169358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.169362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.169918 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.170409 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.170414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.170417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.170627 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.171470 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:17.171475 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:17.171721 (+ 246us) client_negotiation.cc:769] Sending connection context
0929 11:27:17.172965 (+ 1244us) client_negotiation.cc:240] Negotiation successful
0929 11:27:17.173288 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22156,"thread_start_us":13675,"threads_started":1}
I20260929 11:27:17.176059 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.153445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:38757 (local address 127.0.0.1:36396)
0929 11:27:17.153840 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:17.153860 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:17.153968 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:17.158226 (+ 4258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:17.158232 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:17.158678 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:17.158892 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.158904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.160805 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.160811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.162157 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.162173 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.163411 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.163418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.163592 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.166120 (+ 2528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:17.166144 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:17.168696 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:17.173043 (+ 4347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.173049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.173053 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.173620 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.174095 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.174099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.174103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.174399 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.175176 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:17.175182 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:17.175377 (+ 195us) client_negotiation.cc:769] Sending connection context
0929 11:27:17.175530 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 11:27:17.175795 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272}
I20260929 11:27:17.177038 31093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.154163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.23.153.60:38757)
0929 11:27:17.157781 (+ 3618us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:17.157787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:17.157809 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:27:17.157915 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:17.157920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:17.158006 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:17.159083 (+ 1077us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:17.159412 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.160633 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.162416 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.163140 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.165924 (+ 2784us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.169023 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:17.169035 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:17.169042 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:17.169084 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:17.172855 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.173793 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.173797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.173800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.173966 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.174564 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.174569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.174572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.174955 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:17.176446 (+ 1491us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:17.176576 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 11:27:17.176752 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1606,"thread_start_us":2183,"threads_started":1}
I20260929 11:27:17.178439 30336 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.153.60:38757: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8c00c49401624509b0e2c682d3c361fb"
member_type: NON_VOTER
last_known_addr {
host: "127.23.153.60"
port: 38757
}
attrs {
promote: true
}
}
dest_uuid: "bd60a4cf233742daa64ac4c57ea49ea7"
cas_config_opid_index: 6
I20260929 11:27:17.179247 30336 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: NON_VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: true } }
I20260929 11:27:17.180220 31092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.139320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.23.153.1:41733)
0929 11:27:17.155367 (+ 16047us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:17.155377 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:17.155410 (+ 33us) server_negotiation.cc:407] Connection header received
0929 11:27:17.155481 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:17.155489 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:17.155766 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:17.155951 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:17.158695 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.160327 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.162352 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.163468 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.168547 (+ 5079us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.171196 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:17.171227 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:17.171411 (+ 184us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:17.171467 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:17.176164 (+ 4697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.177235 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.177246 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.177249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.177445 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.177981 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.177986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.177989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.178463 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:17.178805 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:17.179203 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 11:27:17.179787 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15841,"thread_start_us":16804,"threads_started":1}
I20260929 11:27:17.181568 30953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 1 FOLLOWER]: Refusing update from remote peer bd60a4cf233742daa64ac4c57ea49ea7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 11:27:17.181877 31034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:17.181984 30520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:17.185171 31034 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 1 FOLLOWER]: Refusing update from remote peer bd60a4cf233742daa64ac4c57ea49ea7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 11:27:17.185803 31065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 11:27:17.186936 31065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: NON_VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:17.190222 31065 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8c00c49401624509b0e2c682d3c361fb (127.23.153.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: NON_VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: true } } }
I20260929 11:27:17.191491 30953 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8c00c49401624509b0e2c682d3c361fb (127.23.153.60) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: NON_VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: true } } }
I20260929 11:27:17.194532 31065 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.153.60:38757
I20260929 11:27:17.195377 30899 master.cc:537] Master@127.23.153.60:38757 shutting down...
I20260929 11:27:17.195530 31065 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: NON_VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: true } } }
I20260929 11:27:17.195485 31064 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: NON_VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: true } } }
I20260929 11:27:17.195825 31065 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:17.195874 31064 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:17.196655 30981 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: NON_VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: true } } }
I20260929 11:27:17.196964 30981 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:27:17.207449 31034 raft_consensus.cc:858] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } }
W20260929 11:27:17.207636 31034 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } }
I20260929 11:27:17.218333 31034 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b8f3ffad32964656969d4daec256606d (127.23.153.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true } } }
I20260929 11:27:17.239503 31101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true } } }
I20260929 11:27:17.240072 31101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 11:27:17.240509 31072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd60a4cf233742daa64ac4c57ea49ea7. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true } } }
I20260929 11:27:17.240754 31072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 11:27:17.240511 31097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: NON_VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: true } } }
I20260929 11:27:17.241201 31097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 11:27:17.242666 31063 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: NON_VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: true } } }
I20260929 11:27:17.242996 31063 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: This master's current role is: LEADER
W20260929 11:27:17.244056 30322 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 11:27:17.244988 30899 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260929 11:27:17.245491 30899 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260929 11:27:17.245669 30899 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: stopping tablet replica
W20260929 11:27:17.251042 30322 consensus_peers.cc:487] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 -> Peer 8c00c49401624509b0e2c682d3c361fb (127.23.153.60:38757): Couldn't send request to peer 8c00c49401624509b0e2c682d3c361fb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 11:27:17.265206 30899 master.cc:552] Master@127.23.153.60:38757 shutdown complete.
I20260929 11:27:17.265333 30899 master_runner.cc:305] Clearing existing system tablet
I20260929 11:27:17.265738 30899 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:17.265852 30899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:17.269954 30899 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260929 11:27:17.272078 31104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:17.272827 30899 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:17.273203 30899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 11:27:17.273353 30899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/wal
uuid: "8c00c49401624509b0e2c682d3c361fb"
format_stamp: "Formatted at 2026-09-29 11:27:16 on dist-test-slave-qk05"
server_key: "a708594878309b4daa801ed291680ce8"
server_key_iv: "6ff453a2c9c4ba52e6f38c9fa1e589eb"
server_key_version: "encryptionkey@0"
I20260929 11:27:17.278740 30899 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:17.287571 30899 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 11:27:17.287801 30899 log.cc:1196] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 11:27:17.288364 30899 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: Deleting consensus metadata
I20260929 11:27:17.288643 30899 master_runner.cc:315] Copying system tablet from 127.23.153.62:41979
I20260929 11:27:17.291857 30899 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: tablet copy: Beginning tablet copy session from remote peer at address 127.23.153.62:41979
I20260929 11:27:17.312155 30870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.292788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50202 (local address 127.23.153.62:41979)
0929 11:27:17.292981 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:17.292995 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:17.293655 (+ 660us) server_negotiation.cc:407] Connection header received
0929 11:27:17.293849 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:17.293855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:17.293937 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:17.294040 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:17.295166 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.296369 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.298390 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.298975 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.299142 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.302341 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:17.302348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:17.302353 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:17.302498 (+ 145us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:17.307629 (+ 5131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.309215 (+ 1586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.309224 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.309228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.309632 (+ 404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.310182 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.310189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.310193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.310670 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:17.310866 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:17.311607 (+ 741us) server_negotiation.cc:299] Negotiation successful
0929 11:27:17.311858 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 11:27:17.312147 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.292746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.0.0.1:50202)
0929 11:27:17.293517 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:17.293543 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:17.293717 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:17.294086 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:17.294091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:17.294684 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:17.294967 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.294980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.296557 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.296563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.298211 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.298230 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.299405 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.299411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.299625 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.300394 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:17.300420 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:17.302013 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:17.307932 (+ 5919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.307944 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.307948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.308970 (+ 1022us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.309809 (+ 839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.309815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.309819 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.310038 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.310822 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:17.310828 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:17.311112 (+ 284us) client_negotiation.cc:769] Sending connection context
0929 11:27:17.311276 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 11:27:17.311663 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":363,"threads_started":1}
I20260929 11:27:17.313875 30356 tablet_copy_service.cc:140] P bd60a4cf233742daa64ac4c57ea49ea7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8c00c49401624509b0e2c682d3c361fb ({username='oryx', principal='oryx/127.23.153.60@KRBTEST.COM'} at 127.0.0.1:50202)
I20260929 11:27:17.314079 30356 tablet_copy_service.cc:161] P bd60a4cf233742daa64ac4c57ea49ea7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8c00c49401624509b0e2c682d3c361fb at {username='oryx', principal='oryx/127.23.153.60@KRBTEST.COM'} at 127.0.0.1:50202: session id = 8c00c49401624509b0e2c682d3c361fb-00000000000000000000000000000000
I20260929 11:27:17.315299 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 30397
I20260929 11:27:17.317818 30356 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 11:27:17.320035 30899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 11:27:17.329032 30899 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: tablet copy: Starting download of 0 data blocks...
I20260929 11:27:17.329589 30899 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: tablet copy: Starting download of 1 WAL segments...
I20260929 11:27:17.333621 30899 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 11:27:17.337049 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:41733
--local_ip_for_outbound_sockets=127.23.153.1
--tserver_master_addrs=127.23.153.62:41979
--webserver_port=35671
--webserver_interface=127.23.153.1
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.1.keytab
--principal=oryx/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.153.62:41979,127.23.153.61:40211
--tserver_master_addrs=127.23.153.62:41979,127.23.153.61:40211,127.23.153.60:38757 with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:27:17.339643 30356 tablet_copy_service.cc:342] P bd60a4cf233742daa64ac4c57ea49ea7: Request end of tablet copy session 8c00c49401624509b0e2c682d3c361fb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.153.60@KRBTEST.COM'} at 127.0.0.1:50202
I20260929 11:27:17.339804 30356 tablet_copy_service.cc:433] P bd60a4cf233742daa64ac4c57ea49ea7: ending tablet copy session 8c00c49401624509b0e2c682d3c361fb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8c00c49401624509b0e2c682d3c361fb
I20260929 11:27:17.354343 30899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:17.357046 31112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:17.360597 31113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:17.373443 31115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:17.373996 30899 server_base.cc:1034] running on GCE node
I20260929 11:27:17.374404 30899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:17.375244 30899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:17.376479 30899 hybrid_clock.cc:648] HybridClock initialized: now 1790681237376420 us; error 47 us; skew 500 ppm
I20260929 11:27:17.376821 30899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:17 dist-test-slave-qk05 krb5kdc[30291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681237, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:17.379117 30899 init.cc:377] Logged in from keytab as oryx/127.23.153.60@KRBTEST.COM (short username oryx)
I20260929 11:27:17.380678 30899 webserver.cc:460] Webserver started at http://127.23.153.60:44575/ using document root <none> and password file <none>
I20260929 11:27:17.381084 30899 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:17.381186 30899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:17.384498 30899 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 11:27:17.386541 31122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:17.387135 30899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 11:27:17.387272 30899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/wal
uuid: "8c00c49401624509b0e2c682d3c361fb"
format_stamp: "Formatted at 2026-09-29 11:27:16 on dist-test-slave-qk05"
server_key: "a708594878309b4daa801ed291680ce8"
server_key_iv: "6ff453a2c9c4ba52e6f38c9fa1e589eb"
server_key_version: "encryptionkey@0"
I20260929 11:27:17.387477 30899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:17.412935 30899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:17.417459 30899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:17.438194 30899 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.60:38757
I20260929 11:27:17.438258 31173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.60:38757 every 8 connection(s)
I20260929 11:27:17.439322 30899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/master-2/data/info.pb
I20260929 11:27:17.444758 31174 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:27:17.449834 31174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: Bootstrap starting.
I20260929 11:27:17.466703 31174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:27:17.467320 31174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: Bootstrap complete.
I20260929 11:27:17.468503 31174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: NON_VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: true } }
I20260929 11:27:17.468844 31174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8c00c49401624509b0e2c682d3c361fb, State: Initialized, Role: LEARNER
I20260929 11:27:17.469132 31174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: NON_VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: true } }
I20260929 11:27:17.470674 31177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: NON_VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: true } } }
I20260929 11:27:17.471158 31177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: This master's current role is: LEARNER
I20260929 11:27:17.470958 31174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:27:17.482191 31188 catalog_manager.cc:1259] Loaded cluster ID: fb7a3e2cc82d4b3ba2e460537a75d64b
I20260929 11:27:17.482854 31188 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: loading cluster ID for follower catalog manager: success
I20260929 11:27:17.491477 31188 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: acquiring CA information for follower catalog manager: success
I20260929 11:27:17.494318 31188 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 11:27:17.721272 31064 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: NON_VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: true } } }
I20260929 11:27:17.721565 31064 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:17.728421 31189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.721602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:38757 (local address 127.0.0.1:59914)
0929 11:27:17.722037 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:17.722053 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:17.722199 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:17.722803 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:17.722808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:17.722836 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:17.723088 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.723099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.724944 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.724951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.726198 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.726215 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.727539 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.727545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.727784 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.727821 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 11:27:17.727909 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 11:27:17.728066 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":512,"threads_started":1}
I20260929 11:27:17.728420 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.721939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.23.153.60:38757)
0929 11:27:17.722498 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:17.722502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:17.722520 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:27:17.722578 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:17.722581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:17.722648 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:17.722762 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:17.723341 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.724780 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.726400 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.727352 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.727491 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.727759 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:17.727950 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 11:27:17.728089 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":327,"threads_started":1}
I20260929 11:27:17.729772 31149 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260929 11:27:17.730585 31177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd60a4cf233742daa64ac4c57ea49ea7. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: NON_VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: true } } }
I20260929 11:27:17.730916 31177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: This master's current role is: LEARNER
I20260929 11:27:17.731352 31064 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7: attempting to promote NON_VOTER 8c00c49401624509b0e2c682d3c361fb to VOTER
I20260929 11:27:17.732021 31064 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false } }
I20260929 11:27:17.734572 30953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 1 FOLLOWER]: Refusing update from remote peer bd60a4cf233742daa64ac4c57ea49ea7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 11:27:17.734581 31149 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 1 LEARNER]: Refusing update from remote peer bd60a4cf233742daa64ac4c57ea49ea7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 11:27:17.735275 31064 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 11:27:17.735525 31065 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 11:27:17.738143 31065 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8c00c49401624509b0e2c682d3c361fb (127.23.153.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false } } }
I20260929 11:27:17.738933 30953 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8c00c49401624509b0e2c682d3c361fb (127.23.153.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false } } }
I20260929 11:27:17.741302 31063 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false } } }
I20260929 11:27:17.741746 31063 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:17.742512 31063 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false } } }
I20260929 11:27:17.742832 31063 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:17.741338 31065 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false } } }
I20260929 11:27:17.743264 31065 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:17.743232 31149 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8c00c49401624509b0e2c682d3c361fb (127.23.153.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false } } }
I20260929 11:27:17.746379 31194 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false } } }
I20260929 11:27:17.746567 31177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false } } }
I20260929 11:27:17.746791 31177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:27:17.746884 31194 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: This master's current role is: FOLLOWER
W20260929 11:27:17.755333 31111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:17.755636 31111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:17.755762 31111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:17.763568 31111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:17.763865 31111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:17.764102 31111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:27:17.776767 31111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.1.keytab
--principal=oryx/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:41733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=35671
--tserver_master_addrs=127.23.153.62:41979,127.23.153.61:40211,127.23.153.60:38757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:17.778198 31111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:17.779660 31111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:17.794917 31197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:17.794811 31198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:17.794853 31200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:17.795626 31111 server_base.cc:1034] running on GCE node
I20260929 11:27:17.796427 31111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:17.797562 31111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:17.798808 31111 hybrid_clock.cc:648] HybridClock initialized: now 1790681237798736 us; error 65 us; skew 500 ppm
I20260929 11:27:17.799242 31111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:17 dist-test-slave-qk05 krb5kdc[30291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681237, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:17.802680 31111 init.cc:377] Logged in from keytab as oryx/127.23.153.1@KRBTEST.COM (short username oryx)
I20260929 11:27:17.804543 31111 webserver.cc:460] Webserver started at http://127.23.153.1:35671/ using document root <none> and password file <none>
I20260929 11:27:17.805305 31111 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:17.805452 31111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:17.812402 31111 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 11:27:17.815894 31207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:17.817914 31111 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:27:17.818133 31111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "35ce55ccb12942328602182facf3ad15"
format_stamp: "Formatted at 2026-09-29 11:27:14 on dist-test-slave-qk05"
server_key: "1b9484b18dde9e024d594510b86c8c40"
server_key_iv: "68ae3195f099b42528d59c1743a42b71"
server_key_version: "encryptionkey@0"
I20260929 11:27:17.819283 31111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:17.835848 31111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:17.840364 31111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:17.840653 31111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:17.841715 31111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:17.843254 31111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:17.843353 31111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:17.843462 31111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:17.843530 31111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:17.878407 31111 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:41733
I20260929 11:27:17.878464 31319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:41733 every 8 connection(s)
I20260929 11:27:17.879966 31111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:27:17.888389 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 31111
I20260929 11:27:17.888823 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 30531
Sep 29 11:27:17 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.153.2@KRBTEST.COM
I20260929 11:27:17.931540 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.889954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.23.153.2:38021)
0929 11:27:17.908753 (+ 18799us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:17.908761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:17.908792 (+ 31us) server_negotiation.cc:407] Connection header received
0929 11:27:17.908859 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:17.908866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:17.909100 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:17.909276 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:17.911159 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.912288 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.913652 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.914264 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.914399 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.921239 (+ 6840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:17.921264 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:17.921286 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:17.921351 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:17.926187 (+ 4836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.927282 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.927292 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.927295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.927484 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.927973 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.927978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.927981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.928727 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:17.930622 (+ 1895us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:17.930816 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 11:27:17.931129 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18605,"thread_start_us":19526,"threads_started":1}
I20260929 11:27:17.933290 30654 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 29 11:27:17 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681237, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.61@KRBTEST.COM
Sep 29 11:27:17 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681237, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.60@KRBTEST.COM
Sep 29 11:27:17 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681237, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM
I20260929 11:27:17.945148 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.883665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:59381 (local address 127.23.153.61:40211)
0929 11:27:17.910119 (+ 26454us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:17.910126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:17.910150 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:27:17.914685 (+ 4535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:17.914691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:17.914799 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:17.914915 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:17.917857 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.919113 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.926470 (+ 7357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.927164 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.927285 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.935934 (+ 8649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:17.935943 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:17.935950 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:17.936003 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:17.940122 (+ 4119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.941974 (+ 1852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.941980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.941984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.942137 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.942617 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.942622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.942626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.943002 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:17.943165 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:17.944088 (+ 923us) server_negotiation.cc:299] Negotiation successful
0929 11:27:17.944678 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26272,"thread_start_us":26477,"threads_started":1}
I20260929 11:27:17.945334 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.884044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:59587 (local address 127.23.153.60:38757)
0929 11:27:17.884240 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:17.884246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:17.914388 (+ 30142us) server_negotiation.cc:407] Connection header received
0929 11:27:17.914650 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:17.914656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:17.914743 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:17.914813 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:17.920081 (+ 5268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.921217 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.928531 (+ 7314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.929171 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.929268 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.936394 (+ 7126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:17.936401 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:17.936406 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:17.936452 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:17.939808 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.940939 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.940944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.940947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.941145 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.941725 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.941730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.941733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.942084 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:17.942234 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:17.944665 (+ 2431us) server_negotiation.cc:299] Negotiation successful
0929 11:27:17.944940 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:27:17.947184 31325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.884402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:38757 (local address 127.23.153.1:59587)
0929 11:27:17.914266 (+ 29864us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:17.914292 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:17.914507 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:17.915445 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:17.915459 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:17.919471 (+ 4012us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:17.919843 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.919860 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.921457 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.921461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.924345 (+ 2884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.924376 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.929574 (+ 5198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.929581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.929787 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.931427 (+ 1640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:17.931455 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:17.936129 (+ 4674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:17.939991 (+ 3862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.940003 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.940027 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.940768 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.941272 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.941278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.941281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.941562 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.942919 (+ 1357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:17.942929 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:17.943583 (+ 654us) client_negotiation.cc:769] Sending connection context
0929 11:27:17.944010 (+ 427us) client_negotiation.cc:240] Negotiation successful
0929 11:27:17.945199 (+ 1189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29684,"mutex_wait_us":626,"spinlock_wait_cycles":69504,"thread_start_us":26436,"threads_started":1}
I20260929 11:27:17.947615 31326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.884402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:40211 (local address 127.23.153.1:59381)
0929 11:27:17.909170 (+ 24768us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:17.909261 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:17.913313 (+ 4052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:17.915444 (+ 2131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:17.915459 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:17.916474 (+ 1015us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:17.917491 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.917526 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.924588 (+ 7062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.924595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.926238 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.926256 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.927636 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.927645 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.928003 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.930287 (+ 2284us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:17.930325 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:17.935579 (+ 5254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:17.940624 (+ 5045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.940631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.940635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.941828 (+ 1193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.942274 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.942279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.942283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.942465 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.943167 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:17.943172 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:17.943554 (+ 382us) client_negotiation.cc:769] Sending connection context
0929 11:27:17.943939 (+ 385us) client_negotiation.cc:240] Negotiation successful
0929 11:27:17.944640 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24337,"mutex_wait_us":29,"spinlock_wait_cycles":7552,"thread_start_us":29379,"threads_started":1}
I20260929 11:27:17.949563 31322 heartbeater.cc:347] Connected to a master server at 127.23.153.61:40211
I20260929 11:27:17.949594 31320 heartbeater.cc:347] Connected to a master server at 127.23.153.60:38757
I20260929 11:27:17.949951 31322 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:17.949957 31320 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:17.950961 31322 heartbeater.cc:511] Master 127.23.153.61:40211 requested a full tablet report, sending...
I20260929 11:27:17.950989 31320 heartbeater.cc:511] Master 127.23.153.60:38757 requested a full tablet report, sending...
I20260929 11:27:17.953397 30943 ts_manager.cc:194] Registered new tserver with Master: 35ce55ccb12942328602182facf3ad15 (127.23.153.1:41733)
I20260929 11:27:17.953553 31139 ts_manager.cc:194] Registered new tserver with Master: 35ce55ccb12942328602182facf3ad15 (127.23.153.1:41733)
I20260929 11:27:17.956739 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.914652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:60319 (local address 127.23.153.62:41979)
0929 11:27:17.933069 (+ 18417us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:17.933076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:17.933098 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:27:17.933171 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:17.933177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:17.933358 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:17.933498 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:17.935529 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.936822 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.938687 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.939378 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.939519 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.946382 (+ 6863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:17.946395 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:17.946399 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:17.946440 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:17.952272 (+ 5832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.954623 (+ 2351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.954630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.954633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.954783 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:17.955315 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:17.955319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:17.955321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:17.955638 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:17.955760 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:17.956182 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 11:27:17.956435 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18289,"thread_start_us":18165,"threads_started":1}
I20260929 11:27:17.956871 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:17.916337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.1:60319)
0929 11:27:17.930589 (+ 14252us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:17.930679 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:17.930970 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:17.933745 (+ 2775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:17.933751 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:17.934308 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:17.935245 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.935264 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.937014 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.937020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.938459 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:17.938484 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:17.939839 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:17.939846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:17.940051 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:17.940725 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:17.940754 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:17.946062 (+ 5308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:17.953898 (+ 7836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.953903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.953907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.954447 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.954899 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:17.954904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:17.954909 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:17.955141 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:17.955760 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:17.955765 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:17.955951 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 11:27:17.956098 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 11:27:17.956425 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13956,"spinlock_wait_cycles":53248,"thread_start_us":13646,"threads_started":1}
I20260929 11:27:17.957759 31321 heartbeater.cc:347] Connected to a master server at 127.23.153.62:41979
I20260929 11:27:17.957955 31321 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:17.958359 31321 heartbeater.cc:511] Master 127.23.153.62:41979 requested a full tablet report, sending...
I20260929 11:27:17.959474 30336 ts_manager.cc:194] Re-registered known tserver with Master: 35ce55ccb12942328602182facf3ad15 (127.23.153.1:41733)
I20260929 11:27:17.960281 30336 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:60319
I20260929 11:27:18.027223 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 30531
I20260929 11:27:18.041451 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:38021
--local_ip_for_outbound_sockets=127.23.153.2
--tserver_master_addrs=127.23.153.62:41979
--webserver_port=40751
--webserver_interface=127.23.153.2
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.2.keytab
--principal=oryx/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.153.62:41979,127.23.153.61:40211
--tserver_master_addrs=127.23.153.62:41979,127.23.153.61:40211,127.23.153.60:38757 with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:18.386852 31332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:18.387233 31332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:18.387390 31332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:18.395678 31332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:18.395849 31332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:18.396028 31332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:27:18.407152 31332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.2.keytab
--principal=oryx/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:38021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=40751
--tserver_master_addrs=127.23.153.62:41979,127.23.153.61:40211,127.23.153.60:38757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:18.408603 31332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:18.410137 31332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:18.424647 31337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:18.424746 31338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:18.424674 31340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:18.425832 31332 server_base.cc:1034] running on GCE node
I20260929 11:27:18.426563 31332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:18.427670 31332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:18.428906 31332 hybrid_clock.cc:648] HybridClock initialized: now 1790681238428865 us; error 44 us; skew 500 ppm
I20260929 11:27:18.429567 31332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:18 dist-test-slave-qk05 krb5kdc[30291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681238, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:18.433907 31332 init.cc:377] Logged in from keytab as oryx/127.23.153.2@KRBTEST.COM (short username oryx)
I20260929 11:27:18.435832 31332 webserver.cc:460] Webserver started at http://127.23.153.2:40751/ using document root <none> and password file <none>
I20260929 11:27:18.436650 31332 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:18.436803 31332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:18.443707 31332 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:27:18.448374 31347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:18.450307 31332 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 11:27:18.450510 31332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "e690037a82ee4b08bcc688edc9073faa"
format_stamp: "Formatted at 2026-09-29 11:27:15 on dist-test-slave-qk05"
server_key: "7ed4f6e5e6e7622fb9b025b3d86735f4"
server_key_iv: "2ddcdbd68b951ea11a63becba7c0f8b3"
server_key_version: "encryptionkey@0"
I20260929 11:27:18.451996 31332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:18.468587 31332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:18.473372 31332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:18.473681 31332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:18.474764 31332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:18.476393 31332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:18.476493 31332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:18.476598 31332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:18.476675 31332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:18.510828 31332 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:38021
I20260929 11:27:18.510881 31459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:38021 every 8 connection(s)
I20260929 11:27:18.512410 31332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:27:18.514252 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 31332
I20260929 11:27:18.514546 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 30665
Sep 29 11:27:18 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.153.3@KRBTEST.COM
Sep 29 11:27:18 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681238, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.60@KRBTEST.COM
Sep 29 11:27:18 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681238, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM
Sep 29 11:27:18 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681238, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.61@KRBTEST.COM
I20260929 11:27:18.558476 31465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:18.515772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55744 (local address 127.23.153.3:40199)
0929 11:27:18.535472 (+ 19700us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:18.535483 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:18.535525 (+ 42us) server_negotiation.cc:407] Connection header received
0929 11:27:18.535616 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:18.535626 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:18.535960 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:18.536186 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:18.538719 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:18.540552 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:18.542304 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:18.543278 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:18.543433 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:18.550654 (+ 7221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:18.550676 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:18.550698 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:18.550745 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:18.554559 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:18.555608 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:18.555620 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:18.555623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:18.555802 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:18.556275 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:18.556280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:18.556283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:18.556769 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:18.557057 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:18.557737 (+ 680us) server_negotiation.cc:299] Negotiation successful
0929 11:27:18.558034 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19500,"thread_start_us":19150,"threads_started":1}
I20260929 11:27:18.560384 30788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:18.561064 31468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:18.517261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:57451 (local address 127.23.153.60:38757)
0929 11:27:18.537582 (+ 20321us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:18.537588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:18.537607 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:27:18.537680 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:18.537684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:18.537756 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:18.540687 (+ 2931us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:18.540892 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:18.542063 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:18.545064 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:18.545684 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:18.545789 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:18.552460 (+ 6671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:18.552469 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:18.552475 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:18.552519 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:18.556551 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:18.557835 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:18.557841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:18.557844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:18.558001 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:18.558598 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:18.558604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:18.558607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:18.558982 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:18.559151 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:18.560253 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0929 11:27:18.560602 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20206,"thread_start_us":20108,"threads_started":1}
I20260929 11:27:18.562237 31466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:18.516469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:38757 (local address 127.23.153.2:57451)
0929 11:27:18.525910 (+ 9441us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:18.525983 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:18.532710 (+ 6727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:18.538301 (+ 5591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:18.538316 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:18.539347 (+ 1031us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:18.540408 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:18.540442 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:18.542341 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:18.542347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:18.544855 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:18.544872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:18.546226 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:18.546235 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:18.546558 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:18.547435 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:18.547469 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:18.552138 (+ 4669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:18.556771 (+ 4633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:18.556782 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:18.556803 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:18.557671 (+ 868us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:18.558148 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:18.558152 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:18.558156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:18.558461 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:18.559263 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:18.559272 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:18.559797 (+ 525us) client_negotiation.cc:769] Sending connection context
0929 11:27:18.560135 (+ 338us) client_negotiation.cc:240] Negotiation successful
0929 11:27:18.560713 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9080,"thread_start_us":8782,"threads_started":1}
I20260929 11:27:18.562865 31469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:18.517564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:52349 (local address 127.23.153.62:41979)
0929 11:27:18.536813 (+ 19249us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:18.536820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:18.536843 (+ 23us) server_negotiation.cc:407] Connection header received
0929 11:27:18.536909 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:18.536916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:18.537000 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:18.537128 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:18.544415 (+ 7287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:18.545606 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:18.547816 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:18.548473 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:18.548592 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:18.553265 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:18.553272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:18.553276 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:18.553322 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:18.557847 (+ 4525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:18.559380 (+ 1533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:18.559386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:18.559389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:18.559561 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:18.560094 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:18.560099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:18.560103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:18.560470 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:18.560619 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:18.562231 (+ 1612us) server_negotiation.cc:299] Negotiation successful
0929 11:27:18.562503 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19110,"thread_start_us":18994,"threads_started":1}
I20260929 11:27:18.563484 31467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:18.516469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.2:52349)
0929 11:27:18.530826 (+ 14357us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:18.530845 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:18.530966 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:18.538301 (+ 7335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:18.538316 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:18.542833 (+ 4517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:18.544207 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:18.544222 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:18.545833 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:18.545839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:18.547601 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:18.547619 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:18.548935 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:18.548942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:18.549127 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:18.549712 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:18.549736 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:18.552957 (+ 3221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:18.558078 (+ 5121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:18.558083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:18.558087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:18.559252 (+ 1165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:18.559695 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:18.559700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:18.559704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:18.559980 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:18.561677 (+ 1697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:18.561684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:18.561880 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 11:27:18.562068 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 11:27:18.562718 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14191,"mutex_wait_us":1083,"spinlock_wait_cycles":66560,"thread_start_us":14741,"threads_started":1}
I20260929 11:27:18.563508 31471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:18.537679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:40211 (local address 127.23.153.2:49587)
0929 11:27:18.543571 (+ 5892us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:18.543592 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:18.543791 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:18.544321 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:18.544326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:18.544912 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:18.545251 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:18.545263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:18.546749 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:18.546753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:18.548377 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:18.548393 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:18.549475 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:18.549481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:18.549679 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:18.550246 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:18.550266 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:18.553107 (+ 2841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:18.557815 (+ 4708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:18.557821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:18.557824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:18.558421 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:18.559718 (+ 1297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:18.559726 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:18.559730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:18.559911 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:18.561313 (+ 1402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:18.561319 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:18.561480 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 11:27:18.561616 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 11:27:18.561920 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5750,"mutex_wait_us":1,"spinlock_wait_cycles":11392,"thread_start_us":5548,"threads_started":1}
I20260929 11:27:18.564122 31460 heartbeater.cc:347] Connected to a master server at 127.23.153.60:38757
I20260929 11:27:18.564709 31461 heartbeater.cc:347] Connected to a master server at 127.23.153.62:41979
I20260929 11:27:18.564736 31470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:18.532156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:49587 (local address 127.23.153.61:40211)
0929 11:27:18.541567 (+ 9411us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:18.541573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:18.543705 (+ 2132us) server_negotiation.cc:407] Connection header received
0929 11:27:18.543924 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:18.543930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:18.544015 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:18.544133 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:18.545441 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:18.546591 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:18.548567 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:18.549162 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:18.549282 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:18.553588 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:18.553595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:18.553599 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:18.553655 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:18.557644 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:18.559046 (+ 1402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:18.559053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:18.559056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:18.559229 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:18.560697 (+ 1468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:18.560702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:18.560704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:18.561077 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:18.561244 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:18.563741 (+ 2497us) server_negotiation.cc:299] Negotiation successful
0929 11:27:18.564366 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9272,"thread_start_us":9158,"threads_started":1}
I20260929 11:27:18.564931 31461 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:18.564937 31460 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:18.565997 31460 heartbeater.cc:511] Master 127.23.153.60:38757 requested a full tablet report, sending...
I20260929 11:27:18.567083 31461 heartbeater.cc:511] Master 127.23.153.62:41979 requested a full tablet report, sending...
I20260929 11:27:18.567723 31462 heartbeater.cc:347] Connected to a master server at 127.23.153.61:40211
I20260929 11:27:18.567862 31462 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:18.568158 30336 ts_manager.cc:194] Re-registered known tserver with Master: e690037a82ee4b08bcc688edc9073faa (127.23.153.2:38021)
I20260929 11:27:18.568315 31462 heartbeater.cc:511] Master 127.23.153.61:40211 requested a full tablet report, sending...
I20260929 11:27:18.568306 31139 ts_manager.cc:194] Registered new tserver with Master: e690037a82ee4b08bcc688edc9073faa (127.23.153.2:38021)
I20260929 11:27:18.569299 30336 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:52349
I20260929 11:27:18.569742 30943 ts_manager.cc:194] Registered new tserver with Master: e690037a82ee4b08bcc688edc9073faa (127.23.153.2:38021)
I20260929 11:27:18.653934 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 30665
I20260929 11:27:18.667876 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:40199
--local_ip_for_outbound_sockets=127.23.153.3
--tserver_master_addrs=127.23.153.62:41979
--webserver_port=34655
--webserver_interface=127.23.153.3
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.3.keytab
--principal=oryx/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.153.62:41979,127.23.153.61:40211
--tserver_master_addrs=127.23.153.62:41979,127.23.153.61:40211,127.23.153.60:38757 with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:27:18.962752 31321 heartbeater.cc:503] Master 127.23.153.62:41979 was elected leader, sending a full tablet report...
W20260929 11:27:19.039325 31473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:19.039639 31473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:19.039745 31473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:19.047216 31473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:19.047376 31473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:19.047509 31473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:27:19.057799 31473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.3.keytab
--principal=oryx/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:40199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=34655
--tserver_master_addrs=127.23.153.62:41979,127.23.153.61:40211,127.23.153.60:38757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:19.059370 31473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:19.060796 31473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:19.075747 31478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:19.075812 31481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:19.075785 31479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:19.076596 31473 server_base.cc:1034] running on GCE node
I20260929 11:27:19.077386 31473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:19.078487 31473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:19.079726 31473 hybrid_clock.cc:648] HybridClock initialized: now 1790681239079660 us; error 53 us; skew 500 ppm
I20260929 11:27:19.080098 31473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:19 dist-test-slave-qk05 krb5kdc[30291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681239, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:19.083530 31473 init.cc:377] Logged in from keytab as oryx/127.23.153.3@KRBTEST.COM (short username oryx)
I20260929 11:27:19.085335 31473 webserver.cc:460] Webserver started at http://127.23.153.3:34655/ using document root <none> and password file <none>
I20260929 11:27:19.086098 31473 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:19.086234 31473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:19.092593 31473 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:27:19.095788 31488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:19.097802 31473 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:27:19.098012 31473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "7ab39c924c1b4797bfac4febda508d47"
format_stamp: "Formatted at 2026-09-29 11:27:15 on dist-test-slave-qk05"
server_key: "2ade45ad72a86b6f4f87bf0b62df5829"
server_key_iv: "79880b2ab5ba56704481c8ceaeee3622"
server_key_version: "encryptionkey@0"
I20260929 11:27:19.099234 31473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:19.112363 31473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:19.117200 31473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:19.117498 31473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:19.118538 31473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:19.120144 31473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:19.120258 31473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:19.120370 31473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:19.120442 31473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:19.155272 31473 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:40199
I20260929 11:27:19.155323 31600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:40199 every 8 connection(s)
I20260929 11:27:19.156817 31473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:27:19.161796 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 31473
Sep 29 11:27:19 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681239, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.61@KRBTEST.COM
Sep 29 11:27:19 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681239, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.60@KRBTEST.COM
Sep 29 11:27:19 dist-test-slave-qk05 krb5kdc[30291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681239, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM
I20260929 11:27:19.207365 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.159944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:53707 (local address 127.23.153.61:40211)
0929 11:27:19.177764 (+ 17820us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:19.177773 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:19.177807 (+ 34us) server_negotiation.cc:407] Connection header received
0929 11:27:19.183164 (+ 5357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:19.183170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:19.183259 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:19.183343 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:19.186235 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.187649 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.191638 (+ 3989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.192274 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.192378 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.197551 (+ 5173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:19.197559 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:19.197564 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:19.197607 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:19.201283 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.202349 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.202356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.202359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.202533 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.204848 (+ 2315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.204854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.204857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.205268 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:19.205439 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:19.206629 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0929 11:27:19.206883 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17656,"thread_start_us":17867,"threads_started":1}
I20260929 11:27:19.210301 31607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.160807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:40211 (local address 127.23.153.3:53707)
0929 11:27:19.171315 (+ 10508us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:19.171400 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:19.182932 (+ 11532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:19.183664 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:19.183670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:19.184560 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:19.185925 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:19.185938 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.190013 (+ 4075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.190028 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:19.191441 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:19.191457 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.192691 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.192697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:19.192866 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.193562 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:19.193595 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:19.197222 (+ 3627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:19.201478 (+ 4256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:19.201489 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:19.201510 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:19.202188 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:19.202703 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:19.202708 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:19.202713 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:19.204691 (+ 1978us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:19.205555 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:19.205566 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:19.206123 (+ 557us) client_negotiation.cc:769] Sending connection context
0929 11:27:19.208095 (+ 1972us) client_negotiation.cc:240] Negotiation successful
0929 11:27:19.208751 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10041,"mutex_wait_us":985,"thread_start_us":9745,"threads_started":1}
I20260929 11:27:19.213742 31603 heartbeater.cc:347] Connected to a master server at 127.23.153.61:40211
I20260929 11:27:19.214344 31603 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:19.215840 31608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.172258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.3:52269)
0929 11:27:19.175352 (+ 3094us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:19.175375 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:19.182240 (+ 6865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:19.183217 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:19.183234 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:19.184563 (+ 1329us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:19.185450 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:19.185473 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.187901 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.187905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:19.190409 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:19.190419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.192115 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.192122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:19.192419 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.194390 (+ 1971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:19.194414 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:19.203788 (+ 9374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:19.209916 (+ 6128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:19.209922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:19.209926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:19.210527 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:19.213602 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:19.213607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:19.213610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:19.213820 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:19.214558 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:19.214563 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:19.214787 (+ 224us) client_negotiation.cc:769] Sending connection context
0929 11:27:19.214997 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 11:27:19.215362 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2668,"mutex_wait_us":385,"thread_start_us":1859,"threads_started":1}
I20260929 11:27:19.216080 31603 heartbeater.cc:511] Master 127.23.153.61:40211 requested a full tablet report, sending...
I20260929 11:27:19.218338 30943 ts_manager.cc:194] Registered new tserver with Master: 7ab39c924c1b4797bfac4febda508d47 (127.23.153.3:40199)
I20260929 11:27:19.219254 31612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.172406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:52269 (local address 127.23.153.62:41979)
0929 11:27:19.179164 (+ 6758us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:19.179172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:19.179211 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:27:19.182535 (+ 3324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:19.182542 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:19.182651 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:19.182776 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:19.185699 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.187747 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.190603 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.191692 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.191808 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.204119 (+ 12311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:19.204126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:19.204131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:19.204174 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:19.208585 (+ 4411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.212467 (+ 3882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.212473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.212477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.212655 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.213969 (+ 1314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.213975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.213978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.214332 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:19.214507 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:19.218513 (+ 4006us) server_negotiation.cc:299] Negotiation successful
0929 11:27:19.218842 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6539,"thread_start_us":6368,"threads_started":1}
I20260929 11:27:19.220789 31602 heartbeater.cc:347] Connected to a master server at 127.23.153.62:41979
I20260929 11:27:19.220960 31602 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:19.221457 31602 heartbeater.cc:511] Master 127.23.153.62:41979 requested a full tablet report, sending...
I20260929 11:27:19.222647 30336 ts_manager.cc:194] Re-registered known tserver with Master: 7ab39c924c1b4797bfac4febda508d47 (127.23.153.3:40199)
I20260929 11:27:19.224077 30336 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:52269
I20260929 11:27:19.227098 31615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.180790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58590 (local address 127.23.153.62:41979)
0929 11:27:19.181297 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:19.181303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:19.195976 (+ 14673us) server_negotiation.cc:407] Connection header received
0929 11:27:19.196191 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:19.196197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:19.196299 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:19.196405 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:19.202973 (+ 6568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.204286 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.210399 (+ 6113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.211080 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.211214 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.218760 (+ 7546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:19.218767 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:19.218771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:19.218809 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:19.223781 (+ 4972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.224846 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.224853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.224856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.225021 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.225419 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.225425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.225429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.225786 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:19.226035 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:19.226454 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 11:27:19.226712 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":822,"threads_started":1}
I20260929 11:27:19.232807 31611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.160807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:38757 (local address 127.23.153.3:51013)
0929 11:27:19.174498 (+ 13691us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:19.174523 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:19.181943 (+ 7420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:19.185732 (+ 3789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:19.185739 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:19.186388 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:19.186691 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:19.186705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.188346 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.188351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:19.190113 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:19.190128 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.196180 (+ 6052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.196187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:19.196395 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.197105 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:19.197141 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:19.200359 (+ 3218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:19.220021 (+ 19662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:19.220026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:19.220030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:19.220602 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:19.227853 (+ 7251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:19.227859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:19.227863 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:19.228069 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:19.231625 (+ 3556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:19.231632 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:19.231835 (+ 203us) client_negotiation.cc:769] Sending connection context
0929 11:27:19.231984 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 11:27:19.232370 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13495,"mutex_wait_us":91,"spinlock_wait_cycles":49024,"thread_start_us":13980,"threads_started":1}
I20260929 11:27:19.236557 31606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.160027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:51013 (local address 127.23.153.60:38757)
0929 11:27:19.185165 (+ 25138us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:19.185201 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:19.185326 (+ 125us) server_negotiation.cc:407] Connection header received
0929 11:27:19.185406 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:19.185412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:19.185505 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:19.186858 (+ 1353us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:19.187045 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.188195 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.193495 (+ 5300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.194146 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.200592 (+ 6446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.200824 (+ 232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:19.200831 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:19.200836 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:19.200867 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:19.212301 (+ 11434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.227156 (+ 14855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.227161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.227164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.227699 (+ 535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.228233 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.228238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.228241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.228609 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:19.235599 (+ 6990us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:19.235782 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 11:27:19.236039 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24883,"thread_start_us":24759,"threads_started":1}
I20260929 11:27:19.237601 31601 heartbeater.cc:347] Connected to a master server at 127.23.153.60:38757
I20260929 11:27:19.237795 31601 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:19.238335 31601 heartbeater.cc:511] Master 127.23.153.60:38757 requested a full tablet report, sending...
I20260929 11:27:19.239720 31139 ts_manager.cc:194] Registered new tserver with Master: 7ab39c924c1b4797bfac4febda508d47 (127.23.153.3:40199)
I20260929 11:27:19.243513 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.196659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.23.153.61:40211)
0929 11:27:19.208002 (+ 11343us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:19.208008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:19.227172 (+ 19164us) server_negotiation.cc:407] Connection header received
0929 11:27:19.227346 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:19.227351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:19.227478 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:19.227584 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:19.229717 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.230788 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.232637 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.233304 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.233415 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.236858 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:19.236866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:19.236870 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:19.236917 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:19.240573 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.241470 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.241476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.241480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.241650 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.242065 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.242070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.242074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.242356 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:19.242584 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:19.242933 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 11:27:19.243192 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11233,"thread_start_us":36401,"threads_started":1}
I20260929 11:27:19.245218 31618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.188542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.23.153.60:38757)
0929 11:27:19.213250 (+ 24708us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:19.213256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:19.213276 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:27:19.213328 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:19.213333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:19.213417 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:19.216934 (+ 3517us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:19.217134 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.218192 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.230857 (+ 12665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.232607 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.232751 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.237955 (+ 5204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:19.237962 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:19.237966 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:19.238007 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:19.241579 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.242607 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.242612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.242616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.242784 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.243354 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.243359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.243362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.243740 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:19.243968 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:19.244388 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 11:27:19.244647 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24448,"thread_start_us":24307,"threads_started":1}
I20260929 11:27:19.238656 30336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:27:19.248626 30336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:27:19.278369 31637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.269128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.23.153.1:41733)
0929 11:27:19.269839 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:19.269847 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:19.269878 (+ 31us) server_negotiation.cc:407] Connection header received
0929 11:27:19.269982 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:19.269990 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:19.270285 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:19.270513 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:19.271233 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.272935 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.275722 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.277229 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.277361 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.277603 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:19.277903 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 11:27:19.278072 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":1010,"threads_started":1}
I20260929 11:27:19.278446 31634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.268345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:41733 (local address 127.0.0.1:32794)
0929 11:27:19.269047 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:19.269062 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:19.269196 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:19.270544 (+ 1348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:19.270548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:19.270575 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:19.270868 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:19.270876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.273211 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.273217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:19.275495 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:19.275513 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.277481 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.277487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:19.277708 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.277737 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 11:27:19.277839 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 11:27:19.277995 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":5,"thread_start_us":446,"threads_started":1}
I20260929 11:27:19.281687 31632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.267643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:40199 (local address 127.0.0.1:55746)
0929 11:27:19.268296 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:19.268316 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:19.268454 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:19.272864 (+ 4410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:19.272868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:19.272894 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:19.273504 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:19.273516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.276063 (+ 2547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.276069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:19.278101 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:19.278118 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.280745 (+ 2627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.280752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:19.280928 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.281003 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 11:27:19.281117 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 11:27:19.281273 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":44,"thread_start_us":338,"threads_started":1}
I20260929 11:27:19.281730 31633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.268000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55746 (local address 127.23.153.3:40199)
0929 11:27:19.271800 (+ 3800us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:19.271807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:19.271836 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:27:19.271916 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:19.271922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:19.272206 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:19.272874 (+ 668us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:19.274217 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.275923 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.279188 (+ 3265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.280528 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.280647 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.280812 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:19.281190 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 11:27:19.281382 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3649,"thread_start_us":2832,"threads_started":1}
I20260929 11:27:19.283213 31636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.268964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51704 (local address 127.23.153.2:38021)
0929 11:27:19.269440 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:19.269448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:19.272318 (+ 2870us) server_negotiation.cc:407] Connection header received
0929 11:27:19.272854 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:19.272862 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:19.273218 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:19.273442 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:19.274189 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.276808 (+ 2619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.280903 (+ 4095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.282212 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.282325 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.282483 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:19.282784 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 11:27:19.282916 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":233,"threads_started":1}
I20260929 11:27:19.283329 31635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.268733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:38021 (local address 127.0.0.1:51704)
0929 11:27:19.271456 (+ 2723us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:19.271474 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:19.272691 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:19.273442 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:19.273450 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:19.273485 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:19.273945 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:19.273966 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.276977 (+ 3011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.276983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:19.279401 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:19.279420 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.282417 (+ 2997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.282424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:19.282610 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.282645 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:27:19.282750 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 11:27:19.282892 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2594,"mutex_wait_us":19,"thread_start_us":2436,"threads_started":1}
I20260929 11:27:19.283259 31255 tablet_service.cc:1467] Processing CreateTablet for tablet 58e65f84813f49079f2263f036606e6d (DEFAULT_TABLE table=test-table [id=b8a9ec2248404230ac4158f90c01e8a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:19.284956 31255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e65f84813f49079f2263f036606e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:19.287551 31395 tablet_service.cc:1467] Processing CreateTablet for tablet 58e65f84813f49079f2263f036606e6d (DEFAULT_TABLE table=test-table [id=b8a9ec2248404230ac4158f90c01e8a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:19.289186 31395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e65f84813f49079f2263f036606e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:19.291280 31536 tablet_service.cc:1467] Processing CreateTablet for tablet 58e65f84813f49079f2263f036606e6d (DEFAULT_TABLE table=test-table [id=b8a9ec2248404230ac4158f90c01e8a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:19.293793 31536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58e65f84813f49079f2263f036606e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:19.300884 31638 tablet_bootstrap.cc:492] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15: Bootstrap starting.
I20260929 11:27:19.305195 31638 tablet_bootstrap.cc:654] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:19.305418 31639 tablet_bootstrap.cc:492] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa: Bootstrap starting.
I20260929 11:27:19.307691 31638 log.cc:824] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:19.309075 31639 tablet_bootstrap.cc:654] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:19.310494 31639 log.cc:824] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:19.310772 31638 tablet_bootstrap.cc:492] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15: No bootstrap required, opened a new log
I20260929 11:27:19.310904 31641 tablet_bootstrap.cc:492] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47: Bootstrap starting.
I20260929 11:27:19.311144 31638 ts_tablet_manager.cc:1397] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260929 11:27:19.315413 31641 tablet_bootstrap.cc:654] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:19.317456 31641 log.cc:824] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:19.318082 31639 tablet_bootstrap.cc:492] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa: No bootstrap required, opened a new log
I20260929 11:27:19.318415 31639 ts_tablet_manager.cc:1397] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260929 11:27:19.321224 31641 tablet_bootstrap.cc:492] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47: No bootstrap required, opened a new log
I20260929 11:27:19.321568 31641 ts_tablet_manager.cc:1397] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260929 11:27:19.321945 31638 raft_consensus.cc:348] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } } peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } }
I20260929 11:27:19.322443 31638 raft_consensus.cc:374] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:19.322557 31638 raft_consensus.cc:729] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ce55ccb12942328602182facf3ad15, State: Initialized, Role: FOLLOWER
I20260929 11:27:19.323581 31638 consensus_queue.cc:260] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } } peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } }
I20260929 11:27:19.329031 31639 raft_consensus.cc:348] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } } peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } }
I20260929 11:27:19.329556 31639 raft_consensus.cc:374] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:19.329654 31639 raft_consensus.cc:729] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e690037a82ee4b08bcc688edc9073faa, State: Initialized, Role: FOLLOWER
I20260929 11:27:19.330339 31639 consensus_queue.cc:260] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } } peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } }
I20260929 11:27:19.331406 31638 ts_tablet_manager.cc:1428] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15: Time spent starting tablet: real 0.020s user 0.013s sys 0.001s
I20260929 11:27:19.332185 31641 raft_consensus.cc:348] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } } peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } }
I20260929 11:27:19.332670 31641 raft_consensus.cc:374] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:19.332777 31641 raft_consensus.cc:729] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab39c924c1b4797bfac4febda508d47, State: Initialized, Role: FOLLOWER
I20260929 11:27:19.333580 31641 consensus_queue.cc:260] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } } peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } }
I20260929 11:27:19.337688 31461 heartbeater.cc:503] Master 127.23.153.62:41979 was elected leader, sending a full tablet report...
I20260929 11:27:19.338449 31639 ts_tablet_manager.cc:1428] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa: Time spent starting tablet: real 0.020s user 0.008s sys 0.012s
I20260929 11:27:19.339190 31602 heartbeater.cc:503] Master 127.23.153.62:41979 was elected leader, sending a full tablet report...
I20260929 11:27:19.339329 31641 ts_tablet_manager.cc:1428] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47: Time spent starting tablet: real 0.018s user 0.008s sys 0.011s
W20260929 11:27:19.384140 31323 tablet.cc:2369] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:27:19.409953 31604 tablet.cc:2369] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:27:19.457332 31645 raft_consensus.cc:482] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:27:19.457639 31645 raft_consensus.cc:504] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } } peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } }
I20260929 11:27:19.459394 31645 leader_election.cc:288] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ab39c924c1b4797bfac4febda508d47 (127.23.153.3:40199), 35ce55ccb12942328602182facf3ad15 (127.23.153.1:41733)
I20260929 11:27:19.468140 31633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.459923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:47943 (local address 127.23.153.3:40199)
0929 11:27:19.460244 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:19.460251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:19.460835 (+ 584us) server_negotiation.cc:407] Connection header received
0929 11:27:19.461376 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:19.461383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:19.461469 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:19.461557 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:19.462651 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.463566 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.465858 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.466881 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.467181 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.467321 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:19.467715 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 11:27:19.467878 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 11:27:19.468155 31637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.461002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:57007 (local address 127.23.153.1:41733)
0929 11:27:19.461422 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:19.461428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:19.461481 (+ 53us) server_negotiation.cc:407] Connection header received
0929 11:27:19.461660 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:19.461665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:19.461738 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:19.461823 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:19.462496 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.463657 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.465871 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.466732 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.466889 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.466950 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:19.467710 (+ 760us) server_negotiation.cc:299] Negotiation successful
0929 11:27:19.467866 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260929 11:27:19.468353 31647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.459771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:40199 (local address 127.23.153.2:47943)
0929 11:27:19.460589 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:19.460674 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:19.460920 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:19.461778 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:19.461783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:19.461810 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:19.462496 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:19.462510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.463853 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.463861 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:19.465653 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:19.465672 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.467257 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.467262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:19.467498 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.467557 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 11:27:19.467660 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 11:27:19.467904 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":146,"thread_start_us":439,"threads_started":1}
I20260929 11:27:19.468482 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.460670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:41733 (local address 127.23.153.2:57007)
0929 11:27:19.461397 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:19.461413 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:19.461525 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:19.461848 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:19.461851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:19.461875 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:19.462282 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:19.462291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.463889 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.463893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:19.465653 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:19.465672 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.466974 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.466981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:19.467497 (+ 516us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.467554 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 11:27:19.467654 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 11:27:19.467904 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":102,"thread_start_us":358,"threads_started":1}
I20260929 11:27:19.469435 31275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e65f84813f49079f2263f036606e6d" candidate_uuid: "e690037a82ee4b08bcc688edc9073faa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ce55ccb12942328602182facf3ad15" is_pre_election: true
I20260929 11:27:19.469439 31556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e65f84813f49079f2263f036606e6d" candidate_uuid: "e690037a82ee4b08bcc688edc9073faa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ab39c924c1b4797bfac4febda508d47" is_pre_election: true
I20260929 11:27:19.469928 31556 raft_consensus.cc:2454] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e690037a82ee4b08bcc688edc9073faa in term 0.
I20260929 11:27:19.469928 31275 raft_consensus.cc:2454] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e690037a82ee4b08bcc688edc9073faa in term 0.
I20260929 11:27:19.470714 31351 leader_election.cc:302] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35ce55ccb12942328602182facf3ad15, e690037a82ee4b08bcc688edc9073faa; no voters:
I20260929 11:27:19.471217 31645 raft_consensus.cc:2781] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:27:19.471370 31645 raft_consensus.cc:482] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:27:19.471487 31645 raft_consensus.cc:3037] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:19.474048 31645 raft_consensus.cc:504] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } } peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } }
I20260929 11:27:19.474699 31645 leader_election.cc:288] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [CANDIDATE]: Term 1 election: Requested vote from peers 7ab39c924c1b4797bfac4febda508d47 (127.23.153.3:40199), 35ce55ccb12942328602182facf3ad15 (127.23.153.1:41733)
I20260929 11:27:19.475497 31556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e65f84813f49079f2263f036606e6d" candidate_uuid: "e690037a82ee4b08bcc688edc9073faa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ab39c924c1b4797bfac4febda508d47"
I20260929 11:27:19.475694 31275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58e65f84813f49079f2263f036606e6d" candidate_uuid: "e690037a82ee4b08bcc688edc9073faa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ce55ccb12942328602182facf3ad15"
I20260929 11:27:19.475740 31556 raft_consensus.cc:3037] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:19.475920 31275 raft_consensus.cc:3037] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:19.478278 31556 raft_consensus.cc:2454] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e690037a82ee4b08bcc688edc9073faa in term 1.
I20260929 11:27:19.478394 31275 raft_consensus.cc:2454] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e690037a82ee4b08bcc688edc9073faa in term 1.
I20260929 11:27:19.478821 31351 leader_election.cc:302] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ab39c924c1b4797bfac4febda508d47, e690037a82ee4b08bcc688edc9073faa; no voters:
I20260929 11:27:19.479189 31645 raft_consensus.cc:2781] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:19.479702 31645 raft_consensus.cc:686] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [term 1 LEADER]: Becoming Leader. State: Replica: e690037a82ee4b08bcc688edc9073faa, State: Running, Role: LEADER
I20260929 11:27:19.480296 31645 consensus_queue.cc:237] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } } peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } }
I20260929 11:27:19.485064 30335 catalog_manager.cc:5495] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa reported cstate change: term changed from 0 to 1, leader changed from <none> to e690037a82ee4b08bcc688edc9073faa (127.23.153.2). New cstate: current_term: 1 leader_uuid: "e690037a82ee4b08bcc688edc9073faa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } health_report { overall_health: HEALTHY } } }
W20260929 11:27:19.533084 31463 tablet.cc:2369] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:27:19.534363 31636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.528674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.23.153.2:38021)
0929 11:27:19.528855 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:19.528859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:19.528962 (+ 103us) server_negotiation.cc:407] Connection header received
0929 11:27:19.529149 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:19.529153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:19.529236 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:19.529378 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:27:19.529694 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.530447 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.531858 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.532299 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.532404 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.533011 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:27:19.533754 (+ 743us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:27:19.533914 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:19.533992 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 11:27:19.534075 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 11:27:19.543779 31275 raft_consensus.cc:1261] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15 [term 1 FOLLOWER]: Refusing update from remote peer e690037a82ee4b08bcc688edc9073faa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:27:19.543779 31556 raft_consensus.cc:1261] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47 [term 1 FOLLOWER]: Refusing update from remote peer e690037a82ee4b08bcc688edc9073faa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:27:19.545161 31645 consensus_queue.cc:1035] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:19.545517 31649 consensus_queue.cc:1035] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:19.560947 31654 mvcc.cc:205] Tried to move back new op lower bound from 7334630357158604800 to 7334630356915458048. Current Snapshot: MvccSnapshot[applied={T|T < 7334630357158604800}]
I20260929 11:27:19.577214 30336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:27:19.577775 30336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:27:19.585446 30336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd60a4cf233742daa64ac4c57ea49ea7: Sending DeleteTablet for 3 replicas of tablet 58e65f84813f49079f2263f036606e6d
I20260929 11:27:19.587402 31536 tablet_service.cc:1514] Processing DeleteTablet for tablet 58e65f84813f49079f2263f036606e6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:19 UTC) from {username='oryx', principal='oryx/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:55746
I20260929 11:27:19.587855 31395 tablet_service.cc:1514] Processing DeleteTablet for tablet 58e65f84813f49079f2263f036606e6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:19 UTC) from {username='oryx', principal='oryx/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:51704
I20260929 11:27:19.588176 31255 tablet_service.cc:1514] Processing DeleteTablet for tablet 58e65f84813f49079f2263f036606e6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:19 UTC) from {username='oryx', principal='oryx/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:32794
I20260929 11:27:19.588348 31662 tablet_replica.cc:330] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47: stopping tablet replica
I20260929 11:27:19.588855 31662 raft_consensus.cc:2229] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:27:19.589434 31662 raft_consensus.cc:2258] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:19.592334 31662 ts_tablet_manager.cc:1905] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:19.600926 31663 tablet_replica.cc:330] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa: stopping tablet replica
I20260929 11:27:19.601459 31663 raft_consensus.cc:2229] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:27:19.602030 31662 ts_tablet_manager.cc:1918] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:27:19.602221 31663 raft_consensus.cc:2258] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:19.602512 31662 log.cc:1196] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/58e65f84813f49079f2263f036606e6d
I20260929 11:27:19.603173 31662 ts_tablet_manager.cc:1939] T 58e65f84813f49079f2263f036606e6d P 7ab39c924c1b4797bfac4febda508d47: Deleting consensus metadata
I20260929 11:27:19.605173 31663 ts_tablet_manager.cc:1905] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:19.605619 30323 catalog_manager.cc:4841] TS 7ab39c924c1b4797bfac4febda508d47 (127.23.153.3:40199): tablet 58e65f84813f49079f2263f036606e6d (table test-table [id=b8a9ec2248404230ac4158f90c01e8a1]) successfully deleted
I20260929 11:27:19.608248 31664 tablet_replica.cc:330] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15: stopping tablet replica
I20260929 11:27:19.608765 31664 raft_consensus.cc:2229] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:27:19.608989 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 30304
I20260929 11:27:19.609368 31664 raft_consensus.cc:2258] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:19.610227 30366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:19.612946 31664 ts_tablet_manager.cc:1905] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:19.620761 31664 ts_tablet_manager.cc:1918] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:27:19.620981 31664 log.cc:1196] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/58e65f84813f49079f2263f036606e6d
I20260929 11:27:19.621589 31664 ts_tablet_manager.cc:1939] T 58e65f84813f49079f2263f036606e6d P 35ce55ccb12942328602182facf3ad15: Deleting consensus metadata
I20260929 11:27:19.628105 31663 ts_tablet_manager.cc:1918] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:27:19.628315 31663 log.cc:1196] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/58e65f84813f49079f2263f036606e6d
I20260929 11:27:19.628906 31663 ts_tablet_manager.cc:1939] T 58e65f84813f49079f2263f036606e6d P e690037a82ee4b08bcc688edc9073faa: Deleting consensus metadata
I20260929 11:27:19.633853 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.591265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38692 (local address 127.23.153.61:40211)
0929 11:27:19.591439 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:19.591445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:19.607684 (+ 16239us) server_negotiation.cc:407] Connection header received
0929 11:27:19.607983 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:19.607989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:19.608079 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:19.608155 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:19.611648 (+ 3493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.612508 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.614993 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.615766 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.615916 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.623200 (+ 7284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:19.623207 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:19.623211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:19.623257 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:19.626636 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.627650 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.627656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.627660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.627931 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.631993 (+ 4062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.632000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.632004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.632446 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:19.632757 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:19.633186 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 11:27:19.633455 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:27:19.652068 31618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.607838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.23.153.60:38757)
0929 11:27:19.608024 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:19.608029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:19.634134 (+ 26105us) server_negotiation.cc:407] Connection header received
0929 11:27:19.634412 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:19.634418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:19.634504 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:19.634579 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:19.636610 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.637572 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.639653 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.640321 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.640426 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.644667 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:19.644676 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:19.644680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:19.644730 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:19.648742 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.649853 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.649860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.649864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.650036 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.650452 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.650457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.650461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.650819 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:19.651111 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:19.651466 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 11:27:19.651726 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 11:27:19.767620 30323 catalog_manager.cc:4841] TS 35ce55ccb12942328602182facf3ad15 (127.23.153.1:41733): tablet 58e65f84813f49079f2263f036606e6d (table test-table [id=b8a9ec2248404230ac4158f90c01e8a1]) successfully deleted
I20260929 11:27:19.768836 30320 catalog_manager.cc:4841] TS e690037a82ee4b08bcc688edc9073faa (127.23.153.2:38021): tablet 58e65f84813f49079f2263f036606e6d (table test-table [id=b8a9ec2248404230ac4158f90c01e8a1]) successfully deleted
I20260929 11:27:19.769834 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 30304
I20260929 11:27:19.813210 31618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.795404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.23.153.60:38757)
0929 11:27:19.795587 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:19.795593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:19.796511 (+ 918us) server_negotiation.cc:407] Connection header received
0929 11:27:19.796799 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:19.796805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:19.796888 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:19.796962 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:19.799303 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.800566 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.802119 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.802618 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.802742 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.807206 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:19.807216 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:19.807222 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:19.807272 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:19.810314 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.811095 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.811100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.811104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.811255 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.811629 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.811634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.811637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.812029 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:19.812267 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:19.812634 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 11:27:19.812878 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 11:27:19.814072 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:19.796777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.23.153.61:40211)
0929 11:27:19.796992 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:19.796997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:19.797756 (+ 759us) server_negotiation.cc:407] Connection header received
0929 11:27:19.797963 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:19.797968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:19.798062 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:19.798178 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:19.800115 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.800910 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.803270 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:19.803975 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:19.804137 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:19.807555 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:19.807563 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:19.807568 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:19.807614 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:19.810749 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.811422 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.811428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.811431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.811581 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:19.811925 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:19.811929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:19.811932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:19.812269 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:19.812507 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:19.812788 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 11:27:19.813025 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 11:27:20.564779 31673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:20.563988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.2:33157)
0929 11:27:20.564487 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:20.564617 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":241,"threads_started":1}
W20260929 11:27:20.565137 31461 heartbeater.cc:644] Failed to heartbeat to 127.23.153.62:41979 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
I20260929 11:27:20.571283 31674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:20.570381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.1:51461)
0929 11:27:20.570892 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:20.571064 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":225,"threads_started":1}
I20260929 11:27:20.572260 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:20.571413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.3:55067)
0929 11:27:20.571875 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:20.572013 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":233,"threads_started":1}
W20260929 11:27:20.572468 31321 heartbeater.cc:644] Failed to heartbeat to 127.23.153.62:41979 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
W20260929 11:27:20.572680 31602 heartbeater.cc:644] Failed to heartbeat to 127.23.153.62:41979 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
I20260929 11:27:21.086374 31676 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bd60a4cf233742daa64ac4c57ea49ea7)
I20260929 11:27:21.086391 31677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bd60a4cf233742daa64ac4c57ea49ea7)
I20260929 11:27:21.086649 31676 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false } }
I20260929 11:27:21.086645 31677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false } }
I20260929 11:27:21.088416 31676 leader_election.cc:288] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd60a4cf233742daa64ac4c57ea49ea7 (127.23.153.62:41979), 8c00c49401624509b0e2c682d3c361fb (127.23.153.60:38757)
I20260929 11:27:21.088572 31677 leader_election.cc:288] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd60a4cf233742daa64ac4c57ea49ea7 (127.23.153.62:41979), b8f3ffad32964656969d4daec256606d (127.23.153.61:40211)
I20260929 11:27:21.103631 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.088550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.0.0.1:58998)
0929 11:27:21.102998 (+ 14448us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.103262 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14324,"thread_start_us":14176,"threads_started":1}
W20260929 11:27:21.104091 30928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
I20260929 11:27:21.103861 31681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.089322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.0.0.1:59010)
0929 11:27:21.103383 (+ 14061us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.103633 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13961,"thread_start_us":13855,"threads_started":1}
W20260929 11:27:21.110638 31124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
I20260929 11:27:21.111686 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.110961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.0.0.1:59026)
0929 11:27:21.111202 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.111410 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260929 11:27:21.112039 30928 leader_election.cc:334] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bd60a4cf233742daa64ac4c57ea49ea7 (127.23.153.62:41979): Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
I20260929 11:27:21.112469 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.089056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:38757 (local address 127.0.0.1:59952)
0929 11:27:21.104016 (+ 14960us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.104034 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:21.104364 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:21.105140 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:21.105147 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:21.105199 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:21.105485 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:21.105497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.107263 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.107270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:21.109651 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:21.109679 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.111689 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.111695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:21.111894 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:21.111929 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:27:21.112009 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 11:27:21.112165 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14866,"thread_start_us":17884,"threads_started":1}
I20260929 11:27:21.113250 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.089035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59952 (local address 127.23.153.60:38757)
0929 11:27:21.102452 (+ 13417us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:21.102459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:21.104296 (+ 1837us) server_negotiation.cc:407] Connection header received
0929 11:27:21.104563 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:21.104571 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:21.104831 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:21.104928 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:21.105691 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.107069 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.109957 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.111339 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.111486 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:21.111592 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:21.112645 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0929 11:27:21.112872 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13232,"spinlock_wait_cycles":5888,"thread_start_us":13440,"threads_started":1}
I20260929 11:27:21.114179 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.113288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.0.0.1:59036)
0929 11:27:21.113629 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.113797 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
I20260929 11:27:21.113853 31149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8f3ffad32964656969d4daec256606d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8c00c49401624509b0e2c682d3c361fb" is_pre_election: true
W20260929 11:27:21.114395 31124 leader_election.cc:334] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bd60a4cf233742daa64ac4c57ea49ea7 (127.23.153.62:41979): Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
I20260929 11:27:21.114645 31149 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f3ffad32964656969d4daec256606d in term 1.
I20260929 11:27:21.115394 30929 leader_election.cc:302] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c00c49401624509b0e2c682d3c361fb, b8f3ffad32964656969d4daec256606d; no voters: bd60a4cf233742daa64ac4c57ea49ea7
I20260929 11:27:21.115850 31676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 11:27:21.115983 31676 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 1 FOLLOWER]: Starting leader election (detected failure of leader bd60a4cf233742daa64ac4c57ea49ea7)
I20260929 11:27:21.116099 31676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 1 FOLLOWER]: Advancing to term 2
I20260929 11:27:21.119053 31681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.103786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:40211 (local address 127.0.0.1:38702)
0929 11:27:21.104834 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.104849 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:21.104967 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:21.111539 (+ 6572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:21.111544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:21.111580 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:21.111808 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:21.111819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.114140 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.114150 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:21.116400 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:21.116416 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.118286 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.118302 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:21.118466 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:21.118497 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 11:27:21.118587 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 11:27:21.118716 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"spinlock_wait_cycles":768,"thread_start_us":6452,"threads_started":1}
I20260929 11:27:21.119284 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.107260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38702 (local address 127.23.153.61:40211)
0929 11:27:21.111194 (+ 3934us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:21.111200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:21.111226 (+ 26us) server_negotiation.cc:407] Connection header received
0929 11:27:21.111293 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:21.111299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:21.111390 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:21.112314 (+ 924us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:21.112612 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.113791 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.116587 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.117908 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.118076 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:21.118203 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:21.118635 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 11:27:21.118798 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3315,"thread_start_us":3430,"threads_started":1}
I20260929 11:27:21.119176 31676 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false } }
I20260929 11:27:21.120002 31676 leader_election.cc:288] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [CANDIDATE]: Term 2 election: Requested vote from peers bd60a4cf233742daa64ac4c57ea49ea7 (127.23.153.62:41979), 8c00c49401624509b0e2c682d3c361fb (127.23.153.60:38757)
I20260929 11:27:21.119940 30953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c00c49401624509b0e2c682d3c361fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b8f3ffad32964656969d4daec256606d" is_pre_election: true
I20260929 11:27:21.120322 30953 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c00c49401624509b0e2c682d3c361fb in current term 2: Already voted for candidate b8f3ffad32964656969d4daec256606d in this term.
I20260929 11:27:21.120608 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.120116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.0.0.1:59038)
0929 11:27:21.120338 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.120426 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 11:27:21.120548 31149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8f3ffad32964656969d4daec256606d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8c00c49401624509b0e2c682d3c361fb"
I20260929 11:27:21.120841 31149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 1 FOLLOWER]: Advancing to term 2
I20260929 11:27:21.121006 31124 leader_election.cc:302] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c00c49401624509b0e2c682d3c361fb; no voters: b8f3ffad32964656969d4daec256606d, bd60a4cf233742daa64ac4c57ea49ea7
I20260929 11:27:21.121619 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.121240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.0.0.1:59052)
0929 11:27:21.121441 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.121507 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260929 11:27:21.121802 30928 leader_election.cc:334] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bd60a4cf233742daa64ac4c57ea49ea7 (127.23.153.62:41979): Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
I20260929 11:27:21.123431 31149 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f3ffad32964656969d4daec256606d in term 2.
I20260929 11:27:21.123668 31677 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 11:27:21.123943 30929 leader_election.cc:302] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c00c49401624509b0e2c682d3c361fb, b8f3ffad32964656969d4daec256606d; no voters: bd60a4cf233742daa64ac4c57ea49ea7
I20260929 11:27:21.124271 31676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 2 FOLLOWER]: Leader election won for term 2
I20260929 11:27:21.124734 31676 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [term 2 LEADER]: Becoming Leader. State: Replica: b8f3ffad32964656969d4daec256606d, State: Running, Role: LEADER
I20260929 11:27:21.125115 31676 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false } }
I20260929 11:27:21.126915 31686 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8f3ffad32964656969d4daec256606d. Latest consensus state: current_term: 2 leader_uuid: "b8f3ffad32964656969d4daec256606d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false } } }
I20260929 11:27:21.127398 31686 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:21.128155 31688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:27:21.132104 31688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:27:21.133083 31688 catalog_manager.cc:1259] Loaded cluster ID: fb7a3e2cc82d4b3ba2e460537a75d64b
I20260929 11:27:21.133172 31688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:27:21.134379 31688 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:27:21.135293 31688 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: Loaded TSK: 0
I20260929 11:27:21.136145 31688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:27:21.169723 30943 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:27:21.179080 30943 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:27:21.183702 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.183243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.0.0.1:59064)
0929 11:27:21.183417 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.183541 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 11:27:21.183808 31149 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [term 2 FOLLOWER]: Refusing update from remote peer b8f3ffad32964656969d4daec256606d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 11:27:21.184829 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.184460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.0.0.1:59080)
0929 11:27:21.184632 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.184704 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 11:27:21.184804 31686 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 11:27:21.185146 30928 consensus_peers.cc:487] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d -> Peer bd60a4cf233742daa64ac4c57ea49ea7 (127.23.153.62:41979): Couldn't send request to peer bd60a4cf233742daa64ac4c57ea49ea7. Status: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 11:27:21.187527 31677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8f3ffad32964656969d4daec256606d. Latest consensus state: current_term: 2 leader_uuid: "b8f3ffad32964656969d4daec256606d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false } } }
I20260929 11:27:21.187830 31677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:27:21.189745 31686 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b8f3ffad32964656969d4daec256606d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false } } }
I20260929 11:27:21.190074 31686 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:21.190398 31677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b8f3ffad32964656969d4daec256606d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd60a4cf233742daa64ac4c57ea49ea7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41979 } } peers { permanent_uuid: "b8f3ffad32964656969d4daec256606d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 40211 } attrs { promote: false } } peers { permanent_uuid: "8c00c49401624509b0e2c682d3c361fb" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 38757 } attrs { promote: false } } }
I20260929 11:27:21.190685 31677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c00c49401624509b0e2c682d3c361fb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:27:21.206926 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.199683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:41733 (local address 127.0.0.1:32810)
0929 11:27:21.200186 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.200202 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:21.200312 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:21.200888 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:21.200893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:21.200918 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:21.201198 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:21.201207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.202416 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.202422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:21.204362 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:21.204378 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.206147 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.206155 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:21.206323 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:21.206350 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 11:27:21.206458 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 11:27:21.206622 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404}
I20260929 11:27:21.206954 31696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.200222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32810 (local address 127.23.153.1:41733)
0929 11:27:21.200627 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:21.200631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:21.200646 (+ 15us) server_negotiation.cc:407] Connection header received
0929 11:27:21.200703 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:21.200707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:21.200761 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:21.200854 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:21.201333 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.202284 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.204535 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.205767 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.205921 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:21.206185 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:21.206499 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 11:27:21.206659 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":235,"threads_started":1}
I20260929 11:27:21.208293 31255 tablet_service.cc:1467] Processing CreateTablet for tablet cbbc595f62ee425680a8804c804b335d (DEFAULT_TABLE table=test-table [id=2cebe33a319748bb9550dcf47cd8e945]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:21.208777 31255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbbc595f62ee425680a8804c804b335d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:21.216025 31699 tablet_bootstrap.cc:492] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15: Bootstrap starting.
I20260929 11:27:21.218073 31699 tablet_bootstrap.cc:654] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:21.219926 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.201686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55750 (local address 127.23.153.3:40199)
0929 11:27:21.213363 (+ 11677us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:21.213368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:21.213388 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:27:21.213459 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:21.213464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:21.213556 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:21.213662 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:21.214232 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.215392 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.217613 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.218686 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.218819 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:21.218946 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:21.219554 (+ 608us) server_negotiation.cc:299] Negotiation successful
0929 11:27:21.219686 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11539,"thread_start_us":12088,"threads_started":1}
I20260929 11:27:21.220100 31697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.201196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:40199 (local address 127.0.0.1:55750)
0929 11:27:21.201805 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.201823 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:21.201930 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:21.213711 (+ 11781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:21.213716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:21.213747 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:21.214074 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:21.214084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.215555 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.215562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:21.217426 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:21.217441 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.219127 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.219135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:21.219352 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:21.219391 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 11:27:21.219507 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 11:27:21.219666 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":48,"thread_start_us":568,"threads_started":1}
I20260929 11:27:21.220940 31678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.199196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:38021 (local address 127.0.0.1:51734)
0929 11:27:21.199567 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.199586 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:21.199697 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:21.212472 (+ 12775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:21.212479 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:21.212524 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:21.212748 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:21.212761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.214308 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.214313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:21.216204 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:21.216222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.219202 (+ 2980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.219208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:21.220418 (+ 1210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:21.220447 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 11:27:21.220533 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 11:27:21.220676 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":994}
I20260929 11:27:21.220888 31536 tablet_service.cc:1467] Processing CreateTablet for tablet cbbc595f62ee425680a8804c804b335d (DEFAULT_TABLE table=test-table [id=2cebe33a319748bb9550dcf47cd8e945]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:21.221058 31695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.199372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51734 (local address 127.23.153.2:38021)
0929 11:27:21.211831 (+ 12459us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:21.211839 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:21.211873 (+ 34us) server_negotiation.cc:407] Connection header received
0929 11:27:21.211977 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:21.211983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:21.212106 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:21.212936 (+ 830us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:21.213132 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.214168 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.216849 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.219050 (+ 2201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.219209 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:21.219328 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:21.220588 (+ 1260us) server_negotiation.cc:299] Negotiation successful
0929 11:27:21.220745 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12253,"thread_start_us":11975,"threads_started":1}
I20260929 11:27:21.221333 31536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbbc595f62ee425680a8804c804b335d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:21.221464 31699 tablet_bootstrap.cc:492] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15: No bootstrap required, opened a new log
I20260929 11:27:21.221630 31699 ts_tablet_manager.cc:1397] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 11:27:21.221791 31395 tablet_service.cc:1467] Processing CreateTablet for tablet cbbc595f62ee425680a8804c804b335d (DEFAULT_TABLE table=test-table [id=2cebe33a319748bb9550dcf47cd8e945]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:21.222225 31395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbbc595f62ee425680a8804c804b335d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:21.222374 31699 raft_consensus.cc:348] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } } peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } }
I20260929 11:27:21.222638 31699 raft_consensus.cc:374] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:21.222728 31699 raft_consensus.cc:729] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ce55ccb12942328602182facf3ad15, State: Initialized, Role: FOLLOWER
I20260929 11:27:21.223035 31699 consensus_queue.cc:260] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } } peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } }
I20260929 11:27:21.224481 31699 ts_tablet_manager.cc:1428] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 11:27:21.231534 31704 tablet_bootstrap.cc:492] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47: Bootstrap starting.
I20260929 11:27:21.233914 31704 tablet_bootstrap.cc:654] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:21.235075 31703 tablet_bootstrap.cc:492] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa: Bootstrap starting.
I20260929 11:27:21.237236 31703 tablet_bootstrap.cc:654] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:21.238090 31702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.225018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.1:44879)
0929 11:27:21.237610 (+ 12592us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.237809 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12420,"threads_started":1}
I20260929 11:27:21.239666 31704 tablet_bootstrap.cc:492] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47: No bootstrap required, opened a new log
I20260929 11:27:21.239724 31702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.239222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.1:59985)
0929 11:27:21.239458 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.239552 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260929 11:27:21.239914 31704 ts_tablet_manager.cc:1397] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 11:27:21.240808 31704 raft_consensus.cc:348] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } } peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } }
I20260929 11:27:21.241071 31704 raft_consensus.cc:374] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:21.241182 31704 raft_consensus.cc:729] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab39c924c1b4797bfac4febda508d47, State: Initialized, Role: FOLLOWER
I20260929 11:27:21.241448 31704 consensus_queue.cc:260] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } } peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } }
I20260929 11:27:21.241734 31322 heartbeater.cc:503] Master 127.23.153.61:40211 was elected leader, sending a full tablet report...
I20260929 11:27:21.242311 31703 tablet_bootstrap.cc:492] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa: No bootstrap required, opened a new log
I20260929 11:27:21.242493 31703 ts_tablet_manager.cc:1397] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20260929 11:27:21.242668 31704 ts_tablet_manager.cc:1428] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 11:27:21.243376 31703 raft_consensus.cc:348] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } } peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } }
I20260929 11:27:21.243659 31703 raft_consensus.cc:374] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:21.243762 31703 raft_consensus.cc:729] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e690037a82ee4b08bcc688edc9073faa, State: Initialized, Role: FOLLOWER
I20260929 11:27:21.244378 31703 consensus_queue.cc:260] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } } peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } }
I20260929 11:27:21.246601 31703 ts_tablet_manager.cc:1428] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260929 11:27:21.248107 31603 heartbeater.cc:503] Master 127.23.153.61:40211 was elected leader, sending a full tablet report...
I20260929 11:27:21.248839 31708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.243772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.3:56957)
0929 11:27:21.247225 (+ 3453us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.247382 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3273,"thread_start_us":1187,"threads_started":1}
I20260929 11:27:21.250541 31708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.249548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.3:40577)
0929 11:27:21.250292 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.250394 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":621}
I20260929 11:27:21.251837 31462 heartbeater.cc:503] Master 127.23.153.61:40211 was elected leader, sending a full tablet report...
I20260929 11:27:21.252187 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.246221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.2:48121)
0929 11:27:21.251623 (+ 5402us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.251780 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5262,"thread_start_us":5095,"threads_started":1}
I20260929 11:27:21.253152 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.252822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.2:48159)
0929 11:27:21.252957 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.253032 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 11:27:21.374888 31701 raft_consensus.cc:482] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:27:21.375128 31701 raft_consensus.cc:504] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } } peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } }
I20260929 11:27:21.376718 31701 leader_election.cc:288] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e690037a82ee4b08bcc688edc9073faa (127.23.153.2:38021), 7ab39c924c1b4797bfac4febda508d47 (127.23.153.3:40199)
I20260929 11:27:21.383304 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.377203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:47227 (local address 127.23.153.3:40199)
0929 11:27:21.377406 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:21.377412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:21.377991 (+ 579us) server_negotiation.cc:407] Connection header received
0929 11:27:21.378150 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:21.378155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:21.378229 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:21.378324 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:21.378839 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.379647 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.381395 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.382185 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.382325 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:21.382395 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:21.382840 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 11:27:21.383046 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 11:27:21.383322 31695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.377203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:33079 (local address 127.23.153.2:38021)
0929 11:27:21.377406 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:21.377412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:21.377682 (+ 270us) server_negotiation.cc:407] Connection header received
0929 11:27:21.377904 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:21.377911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:21.378004 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:21.378096 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:21.378614 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.379428 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.381307 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.382285 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.382460 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:21.382536 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:21.382882 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 11:27:21.383084 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 11:27:21.383342 31702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.377215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:38021 (local address 127.23.153.1:33079)
0929 11:27:21.377550 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.377574 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:21.377745 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:21.378111 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:21.378116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:21.378155 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:21.378470 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:21.378481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.379561 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.379566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:21.381121 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:21.381140 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.382528 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.382532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:21.382723 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:21.382746 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 11:27:21.382837 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 11:27:21.382981 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":40}
I20260929 11:27:21.383931 31711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.377249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:40199 (local address 127.23.153.1:47227)
0929 11:27:21.377868 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.377889 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:21.378023 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:21.378353 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:21.378360 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:21.378379 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:21.378692 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:21.378704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.379778 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.379782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:21.381232 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:21.381248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.382445 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.382453 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:21.382674 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:21.382704 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 11:27:21.382784 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 11:27:21.382931 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":8,"thread_start_us":352,"threads_started":1}
I20260929 11:27:21.384385 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbbc595f62ee425680a8804c804b335d" candidate_uuid: "35ce55ccb12942328602182facf3ad15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e690037a82ee4b08bcc688edc9073faa" is_pre_election: true
I20260929 11:27:21.384546 31556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbbc595f62ee425680a8804c804b335d" candidate_uuid: "35ce55ccb12942328602182facf3ad15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ab39c924c1b4797bfac4febda508d47" is_pre_election: true
I20260929 11:27:21.384764 31556 raft_consensus.cc:2454] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35ce55ccb12942328602182facf3ad15 in term 0.
I20260929 11:27:21.384773 31415 raft_consensus.cc:2454] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35ce55ccb12942328602182facf3ad15 in term 0.
I20260929 11:27:21.385269 31211 leader_election.cc:302] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35ce55ccb12942328602182facf3ad15, 7ab39c924c1b4797bfac4febda508d47; no voters:
I20260929 11:27:21.385677 31701 raft_consensus.cc:2781] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:27:21.385804 31701 raft_consensus.cc:482] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:27:21.385900 31701 raft_consensus.cc:3037] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:21.390043 31701 raft_consensus.cc:504] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } } peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } }
I20260929 11:27:21.390738 31701 leader_election.cc:288] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [CANDIDATE]: Term 1 election: Requested vote from peers e690037a82ee4b08bcc688edc9073faa (127.23.153.2:38021), 7ab39c924c1b4797bfac4febda508d47 (127.23.153.3:40199)
I20260929 11:27:21.391345 31415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbbc595f62ee425680a8804c804b335d" candidate_uuid: "35ce55ccb12942328602182facf3ad15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e690037a82ee4b08bcc688edc9073faa"
I20260929 11:27:21.391351 31556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbbc595f62ee425680a8804c804b335d" candidate_uuid: "35ce55ccb12942328602182facf3ad15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ab39c924c1b4797bfac4febda508d47"
I20260929 11:27:21.391574 31415 raft_consensus.cc:3037] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:21.391569 31556 raft_consensus.cc:3037] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:21.393612 31556 raft_consensus.cc:2454] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35ce55ccb12942328602182facf3ad15 in term 1.
I20260929 11:27:21.393703 31415 raft_consensus.cc:2454] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35ce55ccb12942328602182facf3ad15 in term 1.
I20260929 11:27:21.394178 31211 leader_election.cc:302] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 35ce55ccb12942328602182facf3ad15, 7ab39c924c1b4797bfac4febda508d47, e690037a82ee4b08bcc688edc9073faa; no voters:
I20260929 11:27:21.394555 31701 raft_consensus.cc:2781] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:21.395102 31701 raft_consensus.cc:686] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [term 1 LEADER]: Becoming Leader. State: Replica: 35ce55ccb12942328602182facf3ad15, State: Running, Role: LEADER
I20260929 11:27:21.395541 31701 consensus_queue.cc:237] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } } peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } }
I20260929 11:27:21.396364 31711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.395908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.1:41299)
0929 11:27:21.396093 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.396210 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 11:27:21.399684 30942 catalog_manager.cc:5495] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35ce55ccb12942328602182facf3ad15 (127.23.153.1). New cstate: current_term: 1 leader_uuid: "35ce55ccb12942328602182facf3ad15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35ce55ccb12942328602182facf3ad15" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 } health_report { overall_health: UNKNOWN } } }
I20260929 11:27:21.455621 31696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.449792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32812 (local address 127.23.153.1:41733)
0929 11:27:21.450025 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:21.450031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:21.450049 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:27:21.450136 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:21.450139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:21.450223 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:21.450358 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:27:21.450708 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.451747 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.453378 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.453819 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.453945 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:21.454406 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:27:21.454988 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:27:21.455159 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:21.455258 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 11:27:21.455345 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:27:21.460253 31556 raft_consensus.cc:1261] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47 [term 1 FOLLOWER]: Refusing update from remote peer 35ce55ccb12942328602182facf3ad15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:27:21.460572 31415 raft_consensus.cc:1261] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa [term 1 FOLLOWER]: Refusing update from remote peer 35ce55ccb12942328602182facf3ad15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:27:21.461169 31712 consensus_queue.cc:1035] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ab39c924c1b4797bfac4febda508d47" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:21.461596 31701 consensus_queue.cc:1035] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e690037a82ee4b08bcc688edc9073faa" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:21.464826 31708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.464043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.3:48297)
0929 11:27:21.464474 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.464597 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297}
I20260929 11:27:21.465234 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.464602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.2:34039)
0929 11:27:21.464828 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.465024 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260929 11:27:21.471544 31711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.470807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.1:59079)
0929 11:27:21.471224 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.471353 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298}
I20260929 11:27:21.475821 31708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.475275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.3:51277)
0929 11:27:21.475552 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.475650 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260929 11:27:21.481622 31717 mvcc.cc:205] Tried to move back new op lower bound from 7334630365013028864 to 7334630364760649728. Current Snapshot: MvccSnapshot[applied={T|T < 7334630365013028864}]
I20260929 11:27:21.483095 31711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.482512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.1:40401)
0929 11:27:21.482736 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.482867 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260929 11:27:21.484287 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.483683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.23.153.2:44743)
0929 11:27:21.483897 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.484118 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 11:27:21.490787 30942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:27:21.491101 30942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:27:21.496567 30942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8f3ffad32964656969d4daec256606d: Sending DeleteTablet for 3 replicas of tablet cbbc595f62ee425680a8804c804b335d
I20260929 11:27:21.497579 31395 tablet_service.cc:1514] Processing DeleteTablet for tablet cbbc595f62ee425680a8804c804b335d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:21 UTC) from {username='oryx', principal='oryx/127.23.153.61@KRBTEST.COM'} at 127.0.0.1:51734
I20260929 11:27:21.498298 31536 tablet_service.cc:1514] Processing DeleteTablet for tablet cbbc595f62ee425680a8804c804b335d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:21 UTC) from {username='oryx', principal='oryx/127.23.153.61@KRBTEST.COM'} at 127.0.0.1:55750
I20260929 11:27:21.498337 31255 tablet_service.cc:1514] Processing DeleteTablet for tablet cbbc595f62ee425680a8804c804b335d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:21 UTC) from {username='oryx', principal='oryx/127.23.153.61@KRBTEST.COM'} at 127.0.0.1:32810
I20260929 11:27:21.499037 31725 tablet_replica.cc:330] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47: stopping tablet replica
I20260929 11:27:21.499336 31725 raft_consensus.cc:2229] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:27:21.499704 31725 raft_consensus.cc:2258] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:21.500530 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 31111
I20260929 11:27:21.501103 31725 ts_tablet_manager.cc:1905] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:21.509235 31725 ts_tablet_manager.cc:1918] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:27:21.509550 31725 log.cc:1196] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/cbbc595f62ee425680a8804c804b335d
I20260929 11:27:21.510120 31725 ts_tablet_manager.cc:1939] T cbbc595f62ee425680a8804c804b335d P 7ab39c924c1b4797bfac4febda508d47: Deleting consensus metadata
I20260929 11:27:21.511452 30930 catalog_manager.cc:4841] TS 7ab39c924c1b4797bfac4febda508d47 (127.23.153.3:40199): tablet cbbc595f62ee425680a8804c804b335d (table test-table [id=2cebe33a319748bb9550dcf47cd8e945]) successfully deleted
I20260929 11:27:21.531530 31724 tablet_replica.cc:330] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa: stopping tablet replica
I20260929 11:27:21.531781 31724 raft_consensus.cc:2229] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:27:21.532162 31724 raft_consensus.cc:2258] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:21.533541 31724 ts_tablet_manager.cc:1905] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:21.538605 31726 tablet_replica.cc:330] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15: stopping tablet replica
I20260929 11:27:21.538868 31726 raft_consensus.cc:2229] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:27:21.540230 31726 raft_consensus.cc:2258] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:21.540534 31724 ts_tablet_manager.cc:1918] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:27:21.540688 31724 log.cc:1196] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/cbbc595f62ee425680a8804c804b335d
I20260929 11:27:21.541091 31724 ts_tablet_manager.cc:1939] T cbbc595f62ee425680a8804c804b335d P e690037a82ee4b08bcc688edc9073faa: Deleting consensus metadata
I20260929 11:27:21.541605 31726 ts_tablet_manager.cc:1905] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:21.542167 30927 catalog_manager.cc:4841] TS e690037a82ee4b08bcc688edc9073faa (127.23.153.2:38021): tablet cbbc595f62ee425680a8804c804b335d (table test-table [id=2cebe33a319748bb9550dcf47cd8e945]) successfully deleted
I20260929 11:27:21.550246 31726 ts_tablet_manager.cc:1918] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:27:21.550429 31726 log.cc:1196] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/cbbc595f62ee425680a8804c804b335d
I20260929 11:27:21.550962 31726 ts_tablet_manager.cc:1939] T cbbc595f62ee425680a8804c804b335d P 35ce55ccb12942328602182facf3ad15: Deleting consensus metadata
I20260929 11:27:21.551992 31696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.503320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32820 (local address 127.23.153.1:41733)
0929 11:27:21.503528 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:21.503535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:21.527391 (+ 23856us) server_negotiation.cc:407] Connection header received
0929 11:27:21.527684 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:21.527690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:21.527829 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:21.527926 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:21.532567 (+ 4641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.533782 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.535990 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.536722 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.536838 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:21.544925 (+ 8087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:21.544942 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:21.544959 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:21.544999 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:21.548316 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:21.549182 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:21.549197 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:21.549201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:21.549406 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:21.549848 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:21.549853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:21.549857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:21.550358 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:21.550674 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:21.551275 (+ 601us) server_negotiation.cc:299] Negotiation successful
0929 11:27:21.551506 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:27:21.552258 30930 catalog_manager.cc:4841] TS 35ce55ccb12942328602182facf3ad15 (127.23.153.1:41733): tablet cbbc595f62ee425680a8804c804b335d (table test-table [id=2cebe33a319748bb9550dcf47cd8e945]) successfully deleted
I20260929 11:27:21.553249 31315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:21.665223 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 31111
I20260929 11:27:21.685762 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 31332
I20260929 11:27:21.700662 31695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.686541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.23.153.2:38021)
0929 11:27:21.686731 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:21.686737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:21.686779 (+ 42us) server_negotiation.cc:407] Connection header received
0929 11:27:21.687093 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:21.687100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:21.687184 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:21.687290 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:21.689266 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.690080 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.691873 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.692308 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.692429 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:21.694992 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:21.695007 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:21.695052 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:21.695098 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:21.697910 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:21.698659 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:21.698666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:21.698669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:21.698812 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:21.699177 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:21.699181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:21.699183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:21.699579 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:21.699815 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:21.700037 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 11:27:21.700296 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 11:27:21.701458 31455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:21.769212 31730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.768130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.0.0.1:59086)
0929 11:27:21.768839 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.769048 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":344,"threads_started":1}
I20260929 11:27:21.770709 31730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.770360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41979 (local address 127.0.0.1:59096)
0929 11:27:21.770507 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:21.770582 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:41979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 11:27:21.832420 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 31332
I20260929 11:27:21.851220 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 31473
I20260929 11:27:21.868188 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:21.852025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.23.153.3:40199)
0929 11:27:21.852226 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:21.852233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:21.852257 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:27:21.852376 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:21.852382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:21.852478 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:21.852585 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:21.854297 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.855733 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.857582 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:21.858222 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:21.858363 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:21.861575 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:21.861595 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:21.861613 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:21.861656 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:21.864757 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:21.865561 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:21.865572 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:21.865577 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:21.865774 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:21.866278 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:21.866284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:21.866288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:21.866731 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:21.867175 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:21.867510 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 11:27:21.867749 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:27:21.868840 31596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:21.967460 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 31473
I20260929 11:27:21.982532 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 30799
I20260929 11:27:21.983837 30973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:22.108721 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 30799
I20260929 11:27:22.126608 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 30899
I20260929 11:27:22.127765 31169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:22.253121 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 30899
2026-09-29T11:27:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10150 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:27:22 dist-test-slave-qk05 krb5kdc[31737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:27:22 dist-test-slave-qk05 krb5kdc[31737](info): set up 2 sockets
Sep 29 11:27:22 dist-test-slave-qk05 krb5kdc[31737](info): commencing operation
krb5kdc: starting...
W20260929 11:27:23.296586 24164 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:27:23 dist-test-slave-qk05 krb5kdc[31737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:27:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:27:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.62.keytab.
Entry for principal oryx/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.62.keytab.
I20260929 11:27:23.535346 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:41277
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.62.keytab
--principal=oryx/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:41277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:23.903720 31750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:23.904059 31750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:23.904189 31750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:23.912505 31750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:27:23.912650 31750 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 11:27:23.912696 31750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:23.912775 31750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:23.912844 31750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:27:23.912928 31750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:27:23.923571 31750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:41277
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.62.keytab
--principal=oryx/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:41277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:23.925073 31750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:23.926463 31750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:23.939743 31755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:23.940047 31756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:23.940768 31758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:23.940877 31750 server_base.cc:1034] running on GCE node
I20260929 11:27:23.941808 31750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:23.943377 31750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:23.944651 31750 hybrid_clock.cc:648] HybridClock initialized: now 1790681243944578 us; error 41 us; skew 500 ppm
I20260929 11:27:23.945128 31750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:23 dist-test-slave-qk05 krb5kdc[31737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681243, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:23.948436 31750 init.cc:377] Logged in from keytab as oryx/127.23.153.62@KRBTEST.COM (short username oryx)
I20260929 11:27:23.950230 31750 webserver.cc:460] Webserver started at http://127.23.153.62:46833/ using document root <none> and password file <none>
I20260929 11:27:23.951053 31750 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:23.951171 31750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:23.951507 31750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:23.955233 31750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "85f4546e728a4a1cbe83fa0d789f5c39"
format_stamp: "Formatted at 2026-09-29 11:27:23 on dist-test-slave-qk05"
server_key: "a8dd4a01d6d89dd3e80529b8de5dae6c"
server_key_iv: "3d1713eec14757c96f8da96912318a4d"
server_key_version: "encryptionkey@0"
I20260929 11:27:23.956197 31750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "85f4546e728a4a1cbe83fa0d789f5c39"
format_stamp: "Formatted at 2026-09-29 11:27:23 on dist-test-slave-qk05"
server_key: "a8dd4a01d6d89dd3e80529b8de5dae6c"
server_key_iv: "3d1713eec14757c96f8da96912318a4d"
server_key_version: "encryptionkey@0"
I20260929 11:27:23.961941 31750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:27:23.966058 31765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:23.967618 31750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:27:23.967819 31750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "85f4546e728a4a1cbe83fa0d789f5c39"
format_stamp: "Formatted at 2026-09-29 11:27:23 on dist-test-slave-qk05"
server_key: "a8dd4a01d6d89dd3e80529b8de5dae6c"
server_key_iv: "3d1713eec14757c96f8da96912318a4d"
server_key_version: "encryptionkey@0"
I20260929 11:27:23.968046 31750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:23.984236 31750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:23.988806 31750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:23.989087 31750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:24.011160 31750 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:41277
I20260929 11:27:24.011180 31826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:41277 every 8 connection(s)
I20260929 11:27:24.012609 31750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:27:24.017320 31827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:24.018743 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 31750
I20260929 11:27:24.018940 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:27:24.019593 24164 external_mini_cluster.cc:1428] Setting key 82f7602bfcf2b7f9c22f0392f4778446
I20260929 11:27:24.030979 31827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39: Bootstrap starting.
I20260929 11:27:24.035276 31827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:24.036669 31827 log.cc:824] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39: Log is configured to *not* fsync() on all Append() calls
Sep 29 11:27:24 dist-test-slave-qk05 krb5kdc[31737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM
I20260929 11:27:24.040134 31827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39: No bootstrap required, opened a new log
I20260929 11:27:24.046831 31830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:24.021751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58536 (local address 127.23.153.62:41277)
0929 11:27:24.022621 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:24.022642 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:24.022691 (+ 49us) server_negotiation.cc:407] Connection header received
0929 11:27:24.024589 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:24.024616 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:24.025169 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:24.025564 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:24.028077 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:24.030084 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:24.032034 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:24.032965 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:24.033164 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:24.037962 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:24.037972 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:24.037992 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:24.038037 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:24.041429 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:24.042398 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:24.042409 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:24.042416 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:24.042622 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:24.043090 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:24.043094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:24.043096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:24.043983 (+ 887us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:24.044479 (+ 496us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:24.045143 (+ 664us) server_negotiation.cc:299] Negotiation successful
0929 11:27:24.045624 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":385,"threads_started":1}
I20260929 11:27:24.048390 31827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f4546e728a4a1cbe83fa0d789f5c39" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41277 } }
I20260929 11:27:24.048784 31827 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:24.048895 31827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f4546e728a4a1cbe83fa0d789f5c39, State: Initialized, Role: FOLLOWER
I20260929 11:27:24.049499 31827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f4546e728a4a1cbe83fa0d789f5c39" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41277 } }
I20260929 11:27:24.049765 31827 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:27:24.049911 31827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:27:24.050086 31827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:24.052356 31827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f4546e728a4a1cbe83fa0d789f5c39" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41277 } }
I20260929 11:27:24.052898 31827 leader_election.cc:302] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85f4546e728a4a1cbe83fa0d789f5c39; no voters:
I20260929 11:27:24.053476 31827 leader_election.cc:288] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:27:24.053622 31832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:24.054276 31832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39 [term 1 LEADER]: Becoming Leader. State: Replica: 85f4546e728a4a1cbe83fa0d789f5c39, State: Running, Role: LEADER
I20260929 11:27:24.054841 31832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f4546e728a4a1cbe83fa0d789f5c39" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41277 } }
I20260929 11:27:24.056033 31827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:27:24.058154 31833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85f4546e728a4a1cbe83fa0d789f5c39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f4546e728a4a1cbe83fa0d789f5c39" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41277 } } }
I20260929 11:27:24.058733 31833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:24.059919 31841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:27:24.059993 31834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85f4546e728a4a1cbe83fa0d789f5c39. Latest consensus state: current_term: 1 leader_uuid: "85f4546e728a4a1cbe83fa0d789f5c39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f4546e728a4a1cbe83fa0d789f5c39" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 41277 } } }
I20260929 11:27:24.060319 31834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:24.065372 31841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:27:24.074366 31841 catalog_manager.cc:1347] Generated new cluster ID: 907e90dabd034b60b54dbcfebcf21c70
I20260929 11:27:24.074504 31841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:27:24.090381 31841 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:27:24.093027 31841 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:27:24.112212 31841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39: Generated new TSK 0
I20260929 11:27:24.113421 31841 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.1.keytab.
Entry for principal oryx/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.1.keytab.
I20260929 11:27:24.166002 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:41277
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.1.keytab
--principal=oryx/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:24.526938 31853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:24.527273 31853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:24.527381 31853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:24.535946 31853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:24.536082 31853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:24.536175 31853 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:27:24.536260 31853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:27:24.547129 31853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.1.keytab
--principal=oryx/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.153.62:41277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:24.548540 31853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:24.549921 31853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:24.566879 31858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:24.566876 31859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:24.567121 31861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:24.567900 31853 server_base.cc:1034] running on GCE node
I20260929 11:27:24.568585 31853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:24.569624 31853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:24.570895 31853 hybrid_clock.cc:648] HybridClock initialized: now 1790681244570820 us; error 53 us; skew 500 ppm
I20260929 11:27:24.571264 31853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:24 dist-test-slave-qk05 krb5kdc[31737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681244, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:24.574777 31853 init.cc:377] Logged in from keytab as oryx/127.23.153.1@KRBTEST.COM (short username oryx)
I20260929 11:27:24.576468 31853 webserver.cc:460] Webserver started at http://127.23.153.1:37733/ using document root <none> and password file <none>
I20260929 11:27:24.577221 31853 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:24.577332 31853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:24.577657 31853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:24.580890 31853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "d39a0e45aae94ba9adb9b3bfdb391f36"
format_stamp: "Formatted at 2026-09-29 11:27:24 on dist-test-slave-qk05"
server_key: "f346f64fde794e8b297ab35a869b3e2b"
server_key_iv: "1ce5c6469dc58ae8aa2292457303a277"
server_key_version: "encryptionkey@0"
I20260929 11:27:24.581796 31853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "d39a0e45aae94ba9adb9b3bfdb391f36"
format_stamp: "Formatted at 2026-09-29 11:27:24 on dist-test-slave-qk05"
server_key: "f346f64fde794e8b297ab35a869b3e2b"
server_key_iv: "1ce5c6469dc58ae8aa2292457303a277"
server_key_version: "encryptionkey@0"
I20260929 11:27:24.587746 31853 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 11:27:24.591989 31868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:24.593672 31853 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:27:24.593881 31853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "d39a0e45aae94ba9adb9b3bfdb391f36"
format_stamp: "Formatted at 2026-09-29 11:27:24 on dist-test-slave-qk05"
server_key: "f346f64fde794e8b297ab35a869b3e2b"
server_key_iv: "1ce5c6469dc58ae8aa2292457303a277"
server_key_version: "encryptionkey@0"
I20260929 11:27:24.594116 31853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:24.623096 31853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:24.627761 31853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:24.628043 31853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:24.630852 31853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:24.630951 31853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:24.631112 31853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:24.631173 31853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:27:24 dist-test-slave-qk05 krb5kdc[31737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681244, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM
I20260929 11:27:24.658059 31877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:24.632043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:38675 (local address 127.23.153.62:41277)
0929 11:27:24.632460 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:24.632465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:24.633840 (+ 1375us) server_negotiation.cc:407] Connection header received
0929 11:27:24.636420 (+ 2580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:24.636426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:24.636528 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:24.636637 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:24.639341 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:24.640365 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:24.642762 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:24.643513 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:24.643688 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:24.649508 (+ 5820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:24.649518 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:24.649524 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:24.649576 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:24.653370 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:24.654783 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:24.654788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:24.654791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:24.654940 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:24.655504 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:24.655510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:24.655514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:24.655952 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:24.656168 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:24.657562 (+ 1394us) server_negotiation.cc:299] Negotiation successful
0929 11:27:24.657781 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":220,"threads_started":1}
I20260929 11:27:24.660036 31878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:24.632737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41277 (local address 127.23.153.1:38675)
0929 11:27:24.633649 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:24.633699 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:24.636027 (+ 2328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:24.636955 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:24.636967 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:24.637899 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:24.638988 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:24.639059 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:24.640561 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:24.640567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:24.642522 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:24.642546 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:24.643973 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:24.643993 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:24.644352 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:24.645284 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:24.645322 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:24.649117 (+ 3795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:24.653676 (+ 4559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:24.653692 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:24.653720 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:24.654580 (+ 860us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:24.655095 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:24.655099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:24.655101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:24.655340 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:24.656267 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:24.656277 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:24.656938 (+ 661us) client_negotiation.cc:769] Sending connection context
0929 11:27:24.657380 (+ 442us) client_negotiation.cc:240] Negotiation successful
0929 11:27:24.658094 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":382,"threads_started":1}
I20260929 11:27:24.678658 31853 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:41057
I20260929 11:27:24.678658 31988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:41057 every 8 connection(s)
I20260929 11:27:24.680366 31853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:27:24.681970 31877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:24.666397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:56935 (local address 127.23.153.62:41277)
0929 11:27:24.666623 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:24.666631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:24.667302 (+ 671us) server_negotiation.cc:407] Connection header received
0929 11:27:24.667502 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:24.667506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:24.667609 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:24.667729 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:24.669095 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:24.669814 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:24.672167 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:24.672849 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:24.673002 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:24.675906 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:24.675913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:24.675919 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:24.675966 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:24.678797 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:24.679745 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:24.679750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:24.679753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:24.679894 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:24.680280 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:24.680283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:24.680285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:24.680605 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:24.680733 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:24.681266 (+ 533us) server_negotiation.cc:299] Negotiation successful
0929 11:27:24.681520 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 11:27:24.682178 31964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:24.666343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41277 (local address 127.23.153.1:56935)
0929 11:27:24.667136 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:24.667172 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:24.667369 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:24.667783 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:24.667788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:24.668430 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:24.668895 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:24.668911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:24.669963 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:24.669969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:24.671930 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:24.671957 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:24.673341 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:24.673347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:24.673519 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:24.674368 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:24.674400 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:24.675576 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:24.678957 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:24.678964 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:24.678968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:24.679553 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:24.680019 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:24.680024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:24.680028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:24.680178 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:24.680716 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:24.680722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:24.680954 (+ 232us) client_negotiation.cc:769] Sending connection context
0929 11:27:24.681119 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 11:27:24.681534 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":364,"threads_started":1}
I20260929 11:27:24.682916 31989 heartbeater.cc:347] Connected to a master server at 127.23.153.62:41277
I20260929 11:27:24.683306 31989 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:24.684208 31989 heartbeater.cc:511] Master 127.23.153.62:41277 requested a full tablet report, sending...
I20260929 11:27:24.686517 31782 ts_manager.cc:194] Registered new tserver with Master: d39a0e45aae94ba9adb9b3bfdb391f36 (127.23.153.1:41057)
I20260929 11:27:24.688321 31782 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:38675
I20260929 11:27:24.689510 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 31853
I20260929 11:27:24.689708 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:27:24.690500 24164 external_mini_cluster.cc:1428] Setting key d96cdc65f45364a103509970acb11401
WARNING: no policy specified for oryx/127.23.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.2.keytab.
Entry for principal oryx/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.2.keytab.
I20260929 11:27:24.735621 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:41277
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.2.keytab
--principal=oryx/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:25.097663 31993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:25.097973 31993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:25.098105 31993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:25.105737 31993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:25.105875 31993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:25.105986 31993 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:27:25.106094 31993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:27:25.116285 31993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.2.keytab
--principal=oryx/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.153.62:41277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:25.117609 31993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:25.119042 31993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:25.132661 31998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:25.132714 31999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:25.132627 32001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:25.133671 31993 server_base.cc:1034] running on GCE node
I20260929 11:27:25.134349 31993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:25.135499 31993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:25.137060 31993 hybrid_clock.cc:648] HybridClock initialized: now 1790681245136975 us; error 121 us; skew 500 ppm
I20260929 11:27:25.137396 31993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:25 dist-test-slave-qk05 krb5kdc[31737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681245, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:25.140997 31993 init.cc:377] Logged in from keytab as oryx/127.23.153.2@KRBTEST.COM (short username oryx)
I20260929 11:27:25.142781 31993 webserver.cc:460] Webserver started at http://127.23.153.2:37891/ using document root <none> and password file <none>
I20260929 11:27:25.143642 31993 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:25.143767 31993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:25.144111 31993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:25.147302 31993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "66570f7a510a436fa3b2fbbb0d2a907c"
format_stamp: "Formatted at 2026-09-29 11:27:25 on dist-test-slave-qk05"
server_key: "ebf4323d06947f697458eceea47263dd"
server_key_iv: "d314c9aaa406174f8e4e280e39ac7bd2"
server_key_version: "encryptionkey@0"
I20260929 11:27:25.148175 31993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "66570f7a510a436fa3b2fbbb0d2a907c"
format_stamp: "Formatted at 2026-09-29 11:27:25 on dist-test-slave-qk05"
server_key: "ebf4323d06947f697458eceea47263dd"
server_key_iv: "d314c9aaa406174f8e4e280e39ac7bd2"
server_key_version: "encryptionkey@0"
I20260929 11:27:25.154646 31993 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 11:27:25.159250 32008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:25.160930 31993 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:27:25.161104 31993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "66570f7a510a436fa3b2fbbb0d2a907c"
format_stamp: "Formatted at 2026-09-29 11:27:25 on dist-test-slave-qk05"
server_key: "ebf4323d06947f697458eceea47263dd"
server_key_iv: "d314c9aaa406174f8e4e280e39ac7bd2"
server_key_version: "encryptionkey@0"
I20260929 11:27:25.161375 31993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:25.190267 31993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:25.194572 31993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:25.194821 31993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:25.197556 31993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:25.197674 31993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:25.197816 31993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:25.197878 31993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:27:25 dist-test-slave-qk05 krb5kdc[31737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681245, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM
I20260929 11:27:25.223238 32017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:25.198388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:39293 (local address 127.23.153.62:41277)
0929 11:27:25.198818 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:25.198824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:25.200311 (+ 1487us) server_negotiation.cc:407] Connection header received
0929 11:27:25.203359 (+ 3048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:25.203366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:25.203472 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:25.203583 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:25.206435 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:25.207493 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:25.210744 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:25.211372 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:25.211507 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:25.215833 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:25.215839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:25.215843 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:25.215882 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:25.218443 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:25.219905 (+ 1462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:25.219912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:25.219915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:25.220104 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:25.220738 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:25.220744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:25.220748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:25.221143 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:25.221329 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:25.222418 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0929 11:27:25.222724 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":196,"threads_started":1}
I20260929 11:27:25.224254 32018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:25.199148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41277 (local address 127.23.153.2:39293)
0929 11:27:25.200101 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:25.200159 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:25.202836 (+ 2677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:25.203927 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:25.203939 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:25.204940 (+ 1001us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:25.206139 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:25.206175 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:25.207758 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:25.207766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:25.210478 (+ 2712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:25.210518 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:25.211691 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:25.211701 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:25.211927 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:25.212643 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:25.212667 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:25.215514 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:25.218653 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:25.218765 (+ 112us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:25.218799 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:25.219726 (+ 927us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:25.220243 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:25.220249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:25.220253 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:25.220602 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:25.221296 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:25.221312 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:25.221875 (+ 563us) client_negotiation.cc:769] Sending connection context
0929 11:27:25.222243 (+ 368us) client_negotiation.cc:240] Negotiation successful
0929 11:27:25.222813 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":362,"threads_started":1}
I20260929 11:27:25.247081 32093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:25.231530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41277 (local address 127.23.153.2:44201)
0929 11:27:25.232221 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:25.232244 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:25.232436 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:25.232790 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:25.232796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:25.233498 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:25.233924 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:25.233940 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:25.235069 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:25.235076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:25.237047 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:25.237079 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:25.238395 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:25.238402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:25.238529 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:25.239256 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:25.239275 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:25.240114 (+ 839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:25.244084 (+ 3970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:25.244090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:25.244095 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:25.244703 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:25.245207 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:25.245211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:25.245213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:25.245341 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:25.246112 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:25.246115 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:25.246293 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 11:27:25.246413 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 11:27:25.246696 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":332,"threads_started":1}
I20260929 11:27:25.247200 32017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:25.231552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:44201 (local address 127.23.153.62:41277)
0929 11:27:25.231786 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:25.231794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:25.232359 (+ 565us) server_negotiation.cc:407] Connection header received
0929 11:27:25.232554 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:25.232558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:25.232648 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:25.232739 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:25.234169 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:25.234887 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:25.237310 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:25.237943 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:25.238100 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:25.240403 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:25.240410 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:25.240416 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:25.240464 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:25.243868 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:25.244866 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:25.244873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:25.244877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:25.245060 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:25.245440 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:25.245444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:25.245448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:25.245953 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:25.246144 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:25.246530 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 11:27:25.246805 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 11:27:25.253381 31993 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:38847
I20260929 11:27:25.253413 32128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:38847 every 8 connection(s)
I20260929 11:27:25.254616 31993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:27:25.257140 32129 heartbeater.cc:347] Connected to a master server at 127.23.153.62:41277
I20260929 11:27:25.257405 32129 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:25.258210 32129 heartbeater.cc:511] Master 127.23.153.62:41277 requested a full tablet report, sending...
I20260929 11:27:25.259073 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 31993
I20260929 11:27:25.259229 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:27:25.259748 24164 external_mini_cluster.cc:1428] Setting key c1de18172cbe55435e72c6c48e5849f7
I20260929 11:27:25.260053 31782 ts_manager.cc:194] Registered new tserver with Master: 66570f7a510a436fa3b2fbbb0d2a907c (127.23.153.2:38847)
I20260929 11:27:25.261092 31782 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:39293
WARNING: no policy specified for oryx/127.23.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.3.keytab.
Entry for principal oryx/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.3.keytab.
I20260929 11:27:25.310286 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:41277
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.3.keytab
--principal=oryx/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:25.653841 32133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:25.654173 32133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:25.654276 32133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:25.664249 32133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:25.664391 32133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:25.664479 32133 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 11:27:25.664561 32133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:27:25.675388 32133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.3.keytab
--principal=oryx/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.153.62:41277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:25.677503 32133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:25.679410 32133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 11:27:25.692046 31989 heartbeater.cc:503] Master 127.23.153.62:41277 was elected leader, sending a full tablet report...
W20260929 11:27:25.695155 32138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:25.695232 32141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:25.695256 32139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:25.696596 32133 server_base.cc:1034] running on GCE node
I20260929 11:27:25.697631 32133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:25.699054 32133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:25.700349 32133 hybrid_clock.cc:648] HybridClock initialized: now 1790681245700253 us; error 67 us; skew 500 ppm
I20260929 11:27:25.700914 32133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:25 dist-test-slave-qk05 krb5kdc[31737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681245, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:25.705060 32133 init.cc:377] Logged in from keytab as oryx/127.23.153.3@KRBTEST.COM (short username oryx)
I20260929 11:27:25.706868 32133 webserver.cc:460] Webserver started at http://127.23.153.3:37321/ using document root <none> and password file <none>
I20260929 11:27:25.707865 32133 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:25.708009 32133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:25.708389 32133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:25.712548 32133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "17b9257970fe46b196ae09e375463483"
format_stamp: "Formatted at 2026-09-29 11:27:25 on dist-test-slave-qk05"
server_key: "17f98f6e0438e5ceed19b93a9647907c"
server_key_iv: "d5233228fb863992bb3693d4580f3afe"
server_key_version: "encryptionkey@0"
I20260929 11:27:25.713421 32133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "17b9257970fe46b196ae09e375463483"
format_stamp: "Formatted at 2026-09-29 11:27:25 on dist-test-slave-qk05"
server_key: "17f98f6e0438e5ceed19b93a9647907c"
server_key_iv: "d5233228fb863992bb3693d4580f3afe"
server_key_version: "encryptionkey@0"
I20260929 11:27:25.719995 32133 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 11:27:25.725153 32148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:25.726898 32133 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260929 11:27:25.727139 32133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "17b9257970fe46b196ae09e375463483"
format_stamp: "Formatted at 2026-09-29 11:27:25 on dist-test-slave-qk05"
server_key: "17f98f6e0438e5ceed19b93a9647907c"
server_key_iv: "d5233228fb863992bb3693d4580f3afe"
server_key_version: "encryptionkey@0"
I20260929 11:27:25.727370 32133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:25.744107 32133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:25.748914 32133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:25.749202 32133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:25.752338 32133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:25.752441 32133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:25.752617 32133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:25.752688 32133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:27:25 dist-test-slave-qk05 krb5kdc[31737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681245, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM
I20260929 11:27:25.778959 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:25.752805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:35173 (local address 127.23.153.62:41277)
0929 11:27:25.753540 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:25.753546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:25.754567 (+ 1021us) server_negotiation.cc:407] Connection header received
0929 11:27:25.757011 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:25.757017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:25.757112 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:25.757241 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:25.759425 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:25.760517 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:25.764438 (+ 3921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:25.764999 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:25.765150 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:25.770676 (+ 5526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:25.770685 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:25.770692 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:25.770752 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:25.774819 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:25.775912 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:25.775920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:25.775924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:25.776124 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:25.776717 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:25.776721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:25.776723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:25.777103 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:25.777254 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:25.778274 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0929 11:27:25.778549 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":513,"threads_started":1}
I20260929 11:27:25.780246 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:25.753414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41277 (local address 127.23.153.3:35173)
0929 11:27:25.754357 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:25.754416 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:25.756686 (+ 2270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:25.757529 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:25.757542 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:25.758289 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:25.759225 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:25.759256 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:25.760865 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:25.760874 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:25.764143 (+ 3269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:25.764172 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:25.765463 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:25.765472 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:25.765824 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:25.766741 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:25.766787 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:25.770283 (+ 3496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:25.775118 (+ 4835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:25.775128 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:25.775145 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:25.775694 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:25.776255 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:25.776262 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:25.776266 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:25.776573 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:25.777246 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:25.777255 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:25.777828 (+ 573us) client_negotiation.cc:769] Sending connection context
0929 11:27:25.778147 (+ 319us) client_negotiation.cc:240] Negotiation successful
0929 11:27:25.778785 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":373,"threads_started":1}
I20260929 11:27:25.805049 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:25.790574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:52421 (local address 127.23.153.62:41277)
0929 11:27:25.790780 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:25.790787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:25.791693 (+ 906us) server_negotiation.cc:407] Connection header received
0929 11:27:25.791934 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:25.791938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:25.792007 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:25.792088 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:25.793064 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:25.793867 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:25.795270 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:25.795837 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:25.795962 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:25.798851 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:25.798859 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:25.798864 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:25.798915 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:25.801976 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:25.802748 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:25.802755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:25.802759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:25.802896 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:25.803331 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:25.803336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:25.803340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:25.803851 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:25.804007 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:25.804428 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 11:27:25.804668 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:27:25.805150 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:25.790608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41277 (local address 127.23.153.3:52421)
0929 11:27:25.791515 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:25.791537 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:25.791778 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:25.792135 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:25.792140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:25.792607 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:25.792905 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:25.792916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:25.794020 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:25.794024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:25.795120 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:25.795136 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:25.796037 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:25.796043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:25.796252 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:25.797053 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:25.797080 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:25.798563 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:25.802147 (+ 3584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:25.802153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:25.802158 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:25.802616 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:25.803042 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:25.803048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:25.803052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:25.803217 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:25.803984 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:25.803989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:25.804179 (+ 190us) client_negotiation.cc:769] Sending connection context
0929 11:27:25.804302 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 11:27:25.804708 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":390,"threads_started":1}
I20260929 11:27:25.814548 32133 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:44173
I20260929 11:27:25.814581 32268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:44173 every 8 connection(s)
I20260929 11:27:25.816030 32133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:27:25.818890 32269 heartbeater.cc:347] Connected to a master server at 127.23.153.62:41277
I20260929 11:27:25.819227 32269 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:25.820055 32269 heartbeater.cc:511] Master 127.23.153.62:41277 requested a full tablet report, sending...
I20260929 11:27:25.821776 31781 ts_manager.cc:194] Registered new tserver with Master: 17b9257970fe46b196ae09e375463483 (127.23.153.3:44173)
I20260929 11:27:25.822806 31781 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:35173
I20260929 11:27:25.824416 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 32133
I20260929 11:27:25.824576 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:27:25.825099 24164 external_mini_cluster.cc:1428] Setting key 3dd3a5442e12cfe4c7339310bc6dba56
I20260929 11:27:25.828240 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 11:27:25 dist-test-slave-qk05 krb5kdc[31737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681245, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 11:27:25 dist-test-slave-qk05 krb5kdc[31737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:25 dist-test-slave-qk05 krb5kdc[31737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:26.264153 32129 heartbeater.cc:503] Master 127.23.153.62:41277 was elected leader, sending a full tablet report...
I20260929 11:27:26.825982 32269 heartbeater.cc:503] Master 127.23.153.62:41277 was elected leader, sending a full tablet report...
W20260929 11:27:26.849059 32277 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.976s user 0.002s sys 0.012s
W20260929 11:27:26.850267 32157 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:25.867261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58540 (local address 127.23.153.62:41277)
0929 11:27:25.867479 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:25.867485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:25.867793 (+ 308us) server_negotiation.cc:407] Connection header received
0929 11:27:25.867990 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:25.867996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:25.868063 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:25.868157 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:25.869494 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:25.870224 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:25.871560 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:25.871992 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:25.872120 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:26.849946 (+977826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58540: BlockingRecv error: recv got EOF from 127.0.0.1:58540 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 11:27:26 dist-test-slave-qk05 krb5kdc[31737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681245, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM
I20260929 11:27:26.878350 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:26.854149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58556 (local address 127.23.153.62:41277)
0929 11:27:26.854370 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:26.854376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:26.854712 (+ 336us) server_negotiation.cc:407] Connection header received
0929 11:27:26.854836 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:26.854840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:26.854912 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:26.855346 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:26.856686 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:26.857519 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:26.858973 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:26.859479 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:26.859731 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:26.871173 (+ 11442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:26.871179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:26.871183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:26.871230 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:26.874820 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:26.876108 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:26.876114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:26.876117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:26.876334 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:26.876829 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:26.876833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:26.876836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:26.877165 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:26.877454 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:26.877784 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 11:27:26.878054 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 11:27:26.884289 31781 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:27:26.892032 31781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:27:26.918170 32289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:26.906597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51146 (local address 127.23.153.3:44173)
0929 11:27:26.907250 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:26.907259 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:26.907429 (+ 170us) server_negotiation.cc:407] Connection header received
0929 11:27:26.907748 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:26.907755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:26.907975 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:26.908159 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:26.909054 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:26.910805 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:26.914508 (+ 3703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:26.916744 (+ 2236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:26.916877 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:26.917133 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:26.917428 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 11:27:26.917875 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":294,"threads_started":1}
I20260929 11:27:26.918184 32288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:26.906301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:44173 (local address 127.0.0.1:51146)
0929 11:27:26.907273 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:26.907319 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:26.907525 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:26.908250 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:26.908260 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:26.908302 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:26.908806 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:26.908838 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:26.911117 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:26.911127 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:26.914131 (+ 3004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:26.914158 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:26.916973 (+ 2815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:26.916979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:26.917191 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:26.917233 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 11:27:26.917360 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 11:27:26.917809 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":821,"threads_started":1}
I20260929 11:27:26.919417 32290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:26.908009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:38847 (local address 127.0.0.1:48818)
0929 11:27:26.908854 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:26.908879 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:26.909006 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:26.910687 (+ 1681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:26.910692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:26.910717 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:26.911176 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:26.911196 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:26.913528 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:26.913535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:26.916713 (+ 3178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:26.916730 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:26.918579 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:26.918586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:26.918767 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:26.918801 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 11:27:26.918874 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 11:27:26.919083 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":485,"threads_started":1}
I20260929 11:27:26.920258 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:26.908311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48818 (local address 127.23.153.2:38847)
0929 11:27:26.910024 (+ 1713us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:26.910032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:26.910062 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:27:26.910165 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:26.910172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:26.910445 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:26.911258 (+ 813us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:26.911533 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:26.913118 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:26.916902 (+ 3784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:26.918324 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:26.919324 (+ 1000us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:26.919708 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:26.919788 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 11:27:26.919981 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1551,"thread_start_us":2298,"threads_started":1}
I20260929 11:27:26.920600 32292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:26.908903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:41057 (local address 127.0.0.1:40194)
0929 11:27:26.909452 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:26.909468 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:26.909580 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:26.912175 (+ 2595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:26.912182 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:26.912311 (+ 129us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:26.912810 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:26.912826 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:26.914726 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:26.914730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:26.916348 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:26.916365 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:26.919850 (+ 3485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:26.919857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:26.920033 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:26.920066 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 11:27:26.920146 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 11:27:26.920285 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":656,"threads_started":1}
I20260929 11:27:26.920670 32293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:26.909415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.23.153.1:41057)
0929 11:27:26.911273 (+ 1858us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:26.911281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:26.911310 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:27:26.911403 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:26.911410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:26.911797 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:26.912138 (+ 341us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:26.913030 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:26.914514 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:26.916547 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:26.919322 (+ 2775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:26.919487 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:26.919693 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:26.920217 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 11:27:26.920393 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1692,"thread_start_us":1511,"threads_started":1}
I20260929 11:27:26.923676 32199 tablet_service.cc:1467] Processing CreateTablet for tablet c4396ef76b3446e092e710a4cf39fc05 (DEFAULT_TABLE table=test-table [id=a244e0900c43456880ec43d3a8415edb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:26.924551 32059 tablet_service.cc:1467] Processing CreateTablet for tablet c4396ef76b3446e092e710a4cf39fc05 (DEFAULT_TABLE table=test-table [id=a244e0900c43456880ec43d3a8415edb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:26.924551 31919 tablet_service.cc:1467] Processing CreateTablet for tablet c4396ef76b3446e092e710a4cf39fc05 (DEFAULT_TABLE table=test-table [id=a244e0900c43456880ec43d3a8415edb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:26.925439 32199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4396ef76b3446e092e710a4cf39fc05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:26.925944 32059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4396ef76b3446e092e710a4cf39fc05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:26.925953 31919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4396ef76b3446e092e710a4cf39fc05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:26.945032 32294 tablet_bootstrap.cc:492] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483: Bootstrap starting.
I20260929 11:27:26.946251 32296 tablet_bootstrap.cc:492] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36: Bootstrap starting.
I20260929 11:27:26.947235 32295 tablet_bootstrap.cc:492] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c: Bootstrap starting.
I20260929 11:27:26.950294 32296 tablet_bootstrap.cc:654] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:26.950778 32295 tablet_bootstrap.cc:654] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:26.950766 32294 tablet_bootstrap.cc:654] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:26.951635 32296 log.cc:824] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:26.952092 32294 log.cc:824] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:26.952160 32295 log.cc:824] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:26.955902 32294 tablet_bootstrap.cc:492] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483: No bootstrap required, opened a new log
I20260929 11:27:26.955930 32295 tablet_bootstrap.cc:492] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c: No bootstrap required, opened a new log
I20260929 11:27:26.956220 32294 ts_tablet_manager.cc:1397] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 11:27:26.956238 32295 ts_tablet_manager.cc:1397] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.006s
I20260929 11:27:26.957500 32296 tablet_bootstrap.cc:492] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36: No bootstrap required, opened a new log
I20260929 11:27:26.957818 32296 ts_tablet_manager.cc:1397] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260929 11:27:26.965262 32294 raft_consensus.cc:348] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } }
I20260929 11:27:26.965909 32294 raft_consensus.cc:374] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:26.966017 32294 raft_consensus.cc:729] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b9257970fe46b196ae09e375463483, State: Initialized, Role: FOLLOWER
I20260929 11:27:26.966706 32294 consensus_queue.cc:260] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } }
I20260929 11:27:26.967347 32295 raft_consensus.cc:348] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } }
I20260929 11:27:26.967835 32295 raft_consensus.cc:374] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:26.967947 32295 raft_consensus.cc:729] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66570f7a510a436fa3b2fbbb0d2a907c, State: Initialized, Role: FOLLOWER
I20260929 11:27:26.968528 32294 ts_tablet_manager.cc:1428] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260929 11:27:26.968618 32295 consensus_queue.cc:260] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } }
I20260929 11:27:26.970465 32295 ts_tablet_manager.cc:1428] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260929 11:27:26.970294 32296 raft_consensus.cc:348] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } }
I20260929 11:27:26.970844 32296 raft_consensus.cc:374] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:26.970960 32296 raft_consensus.cc:729] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d39a0e45aae94ba9adb9b3bfdb391f36, State: Initialized, Role: FOLLOWER
I20260929 11:27:26.971693 32296 consensus_queue.cc:260] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } }
I20260929 11:27:26.974826 32296 ts_tablet_manager.cc:1428] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
W20260929 11:27:27.007845 32130 tablet.cc:2369] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:27:27.068758 32270 tablet.cc:2369] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:27:27.190059 31990 tablet.cc:2369] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:27:27.194743 32300 raft_consensus.cc:482] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:27:27.195071 32300 raft_consensus.cc:504] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } }
I20260929 11:27:27.196751 32300 leader_election.cc:288] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66570f7a510a436fa3b2fbbb0d2a907c (127.23.153.2:38847), d39a0e45aae94ba9adb9b3bfdb391f36 (127.23.153.1:41057)
I20260929 11:27:27.203689 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:27.197171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:39059 (local address 127.23.153.2:38847)
0929 11:27:27.197399 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:27.197404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:27.197981 (+ 577us) server_negotiation.cc:407] Connection header received
0929 11:27:27.198193 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:27.198199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:27.198291 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:27.198400 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:27.199246 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.200033 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.201892 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.202705 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.202866 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:27.202944 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:27.203319 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 11:27:27.203458 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:27:27.203783 32303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:27.197167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:38847 (local address 127.23.153.3:39059)
0929 11:27:27.197849 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:27.197868 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:27.198014 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:27.198642 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:27.198649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:27.198689 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:27.199085 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:27.199098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.200165 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.200171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:27.201690 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:27.201708 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.202931 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.202937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:27.203151 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:27.203178 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 11:27:27.203269 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 11:27:27.203428 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":352,"threads_started":1}
I20260929 11:27:27.204025 32293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:27.198305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:33227 (local address 127.23.153.1:41057)
0929 11:27:27.198521 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:27.198527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:27.198932 (+ 405us) server_negotiation.cc:407] Connection header received
0929 11:27:27.199143 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:27.199147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:27.199234 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:27.199314 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:27.199866 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.200574 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.202313 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.203141 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.203294 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:27.203369 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:27.203666 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 11:27:27.203795 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 11:27:27.204330 32304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:27.198019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:41057 (local address 127.23.153.3:33227)
0929 11:27:27.198828 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:27.198847 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:27.198985 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:27.199358 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:27.199363 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:27.199389 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:27.199701 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:27.199711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.200726 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.200732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:27.202140 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:27.202155 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.203374 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.203381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:27.203517 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:27.203541 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 11:27:27.203626 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 11:27:27.203762 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":506,"threads_started":1}
I20260929 11:27:27.204928 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4396ef76b3446e092e710a4cf39fc05" candidate_uuid: "17b9257970fe46b196ae09e375463483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" is_pre_election: true
I20260929 11:27:27.205269 31939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4396ef76b3446e092e710a4cf39fc05" candidate_uuid: "17b9257970fe46b196ae09e375463483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" is_pre_election: true
I20260929 11:27:27.205433 32081 raft_consensus.cc:2454] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17b9257970fe46b196ae09e375463483 in term 0.
I20260929 11:27:27.205718 31939 raft_consensus.cc:2454] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17b9257970fe46b196ae09e375463483 in term 0.
I20260929 11:27:27.206128 32150 leader_election.cc:302] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b9257970fe46b196ae09e375463483, 66570f7a510a436fa3b2fbbb0d2a907c; no voters:
I20260929 11:27:27.206537 32300 raft_consensus.cc:2781] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:27:27.206689 32300 raft_consensus.cc:482] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:27:27.206784 32300 raft_consensus.cc:3037] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:27.209091 32300 raft_consensus.cc:504] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } }
I20260929 11:27:27.209694 32300 leader_election.cc:288] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [CANDIDATE]: Term 1 election: Requested vote from peers 66570f7a510a436fa3b2fbbb0d2a907c (127.23.153.2:38847), d39a0e45aae94ba9adb9b3bfdb391f36 (127.23.153.1:41057)
I20260929 11:27:27.210285 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4396ef76b3446e092e710a4cf39fc05" candidate_uuid: "17b9257970fe46b196ae09e375463483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66570f7a510a436fa3b2fbbb0d2a907c"
I20260929 11:27:27.210482 32081 raft_consensus.cc:3037] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:27.210625 31939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4396ef76b3446e092e710a4cf39fc05" candidate_uuid: "17b9257970fe46b196ae09e375463483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36"
I20260929 11:27:27.210829 31939 raft_consensus.cc:3037] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:27.212787 32081 raft_consensus.cc:2454] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17b9257970fe46b196ae09e375463483 in term 1.
I20260929 11:27:27.213025 31939 raft_consensus.cc:2454] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17b9257970fe46b196ae09e375463483 in term 1.
I20260929 11:27:27.213282 32150 leader_election.cc:302] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b9257970fe46b196ae09e375463483, 66570f7a510a436fa3b2fbbb0d2a907c; no voters:
I20260929 11:27:27.213570 32300 raft_consensus.cc:2781] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:27.213982 32300 raft_consensus.cc:686] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [term 1 LEADER]: Becoming Leader. State: Replica: 17b9257970fe46b196ae09e375463483, State: Running, Role: LEADER
I20260929 11:27:27.214509 32300 consensus_queue.cc:237] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } }
I20260929 11:27:27.219376 31782 catalog_manager.cc:5495] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17b9257970fe46b196ae09e375463483 (127.23.153.3). New cstate: current_term: 1 leader_uuid: "17b9257970fe46b196ae09e375463483" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } health_report { overall_health: UNKNOWN } } }
I20260929 11:27:27.295482 32289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:27.290081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51158 (local address 127.23.153.3:44173)
0929 11:27:27.290259 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:27.290264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:27.290285 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:27:27.290389 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:27.290393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:27.290486 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:27.290623 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:27:27.291059 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.291792 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.293268 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.293697 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.293798 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:27.294233 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:27:27.294816 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:27:27.294929 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:27.295180 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 11:27:27.295264 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:27:27.305670 32081 raft_consensus.cc:1261] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 FOLLOWER]: Refusing update from remote peer 17b9257970fe46b196ae09e375463483: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:27:27.305670 31939 raft_consensus.cc:1261] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 FOLLOWER]: Refusing update from remote peer 17b9257970fe46b196ae09e375463483: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:27:27.306893 32305 consensus_queue.cc:1035] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:27.307341 32300 consensus_queue.cc:1035] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:27.339118 31780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:27:27.339422 31780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:27:27.344357 31780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39: Sending DeleteTablet for 3 replicas of tablet c4396ef76b3446e092e710a4cf39fc05
I20260929 11:27:27.345675 32199 tablet_service.cc:1514] Processing DeleteTablet for tablet c4396ef76b3446e092e710a4cf39fc05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:27 UTC) from {username='oryx', principal='oryx/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:51146
I20260929 11:27:27.345790 32059 tablet_service.cc:1514] Processing DeleteTablet for tablet c4396ef76b3446e092e710a4cf39fc05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:27 UTC) from {username='oryx', principal='oryx/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:48818
I20260929 11:27:27.346443 31919 tablet_service.cc:1514] Processing DeleteTablet for tablet c4396ef76b3446e092e710a4cf39fc05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:27 UTC) from {username='oryx', principal='oryx/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:40194
I20260929 11:27:27.347184 31780 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:27:27.347939 31780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:27:27.350551 32320 tablet_replica.cc:330] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36: stopping tablet replica
I20260929 11:27:27.351245 32320 raft_consensus.cc:2229] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:27:27.352034 32320 raft_consensus.cc:2258] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:27.354506 32320 ts_tablet_manager.cc:1905] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:27.359763 32058 tablet_service.cc:1467] Processing CreateTablet for tablet 24b6f5b983054da29f55c1bbdcfaa83f (DEFAULT_TABLE table=test-table [id=6e97b4b2e79e4325a917ad4626d163dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:27.359877 32198 tablet_service.cc:1467] Processing CreateTablet for tablet 24b6f5b983054da29f55c1bbdcfaa83f (DEFAULT_TABLE table=test-table [id=6e97b4b2e79e4325a917ad4626d163dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:27.360430 32058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24b6f5b983054da29f55c1bbdcfaa83f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:27.360426 32198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24b6f5b983054da29f55c1bbdcfaa83f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:27.360925 31919 tablet_service.cc:1467] Processing CreateTablet for tablet 24b6f5b983054da29f55c1bbdcfaa83f (DEFAULT_TABLE table=test-table [id=6e97b4b2e79e4325a917ad4626d163dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:27.361469 31919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24b6f5b983054da29f55c1bbdcfaa83f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:27.364810 32318 tablet_replica.cc:330] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483: stopping tablet replica
I20260929 11:27:27.365355 32318 raft_consensus.cc:2229] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:27:27.366109 32318 raft_consensus.cc:2258] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:27.368440 32294 tablet_bootstrap.cc:492] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483: Bootstrap starting.
I20260929 11:27:27.369387 32295 tablet_bootstrap.cc:492] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c: Bootstrap starting.
I20260929 11:27:27.370266 32294 tablet_bootstrap.cc:654] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:27.371001 32295 tablet_bootstrap.cc:654] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:27.371474 32296 tablet_bootstrap.cc:492] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36: Bootstrap starting.
I20260929 11:27:27.373193 32296 tablet_bootstrap.cc:654] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:27.379609 32295 tablet_bootstrap.cc:492] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c: No bootstrap required, opened a new log
I20260929 11:27:27.380227 32295 ts_tablet_manager.cc:1397] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.001s
I20260929 11:27:27.382309 32295 raft_consensus.cc:348] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:27.384502 32318 ts_tablet_manager.cc:1905] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:27.384799 32295 raft_consensus.cc:374] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:27.384976 32295 raft_consensus.cc:729] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66570f7a510a436fa3b2fbbb0d2a907c, State: Initialized, Role: FOLLOWER
I20260929 11:27:27.385377 32295 consensus_queue.cc:260] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:27.389376 32295 ts_tablet_manager.cc:1428] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20260929 11:27:27.390179 32319 tablet_replica.cc:330] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c: stopping tablet replica
I20260929 11:27:27.392777 32296 tablet_bootstrap.cc:492] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36: No bootstrap required, opened a new log
I20260929 11:27:27.393069 32296 ts_tablet_manager.cc:1397] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.014s
I20260929 11:27:27.394073 32296 raft_consensus.cc:348] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:27.394345 32319 raft_consensus.cc:2229] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:27:27.394420 32296 raft_consensus.cc:374] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:27.394701 32296 raft_consensus.cc:729] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d39a0e45aae94ba9adb9b3bfdb391f36, State: Initialized, Role: FOLLOWER
I20260929 11:27:27.394896 32319 raft_consensus.cc:2258] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:27.394443 32320 ts_tablet_manager.cc:1918] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:27:27.395087 32320 log.cc:1196] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/c4396ef76b3446e092e710a4cf39fc05
I20260929 11:27:27.396104 32320 ts_tablet_manager.cc:1939] T c4396ef76b3446e092e710a4cf39fc05 P d39a0e45aae94ba9adb9b3bfdb391f36: Deleting consensus metadata
I20260929 11:27:27.395366 32296 consensus_queue.cc:260] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:27.399212 32319 ts_tablet_manager.cc:1905] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:27.402730 32296 ts_tablet_manager.cc:1428] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20260929 11:27:27.406419 31767 catalog_manager.cc:4841] TS d39a0e45aae94ba9adb9b3bfdb391f36 (127.23.153.1:41057): tablet c4396ef76b3446e092e710a4cf39fc05 (table test-table [id=a244e0900c43456880ec43d3a8415edb]) successfully deleted
I20260929 11:27:27.408821 32319 ts_tablet_manager.cc:1918] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:27:27.409024 32319 log.cc:1196] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/c4396ef76b3446e092e710a4cf39fc05
I20260929 11:27:27.409672 32319 ts_tablet_manager.cc:1939] T c4396ef76b3446e092e710a4cf39fc05 P 66570f7a510a436fa3b2fbbb0d2a907c: Deleting consensus metadata
I20260929 11:27:27.412091 31767 catalog_manager.cc:4841] TS 66570f7a510a436fa3b2fbbb0d2a907c (127.23.153.2:38847): tablet c4396ef76b3446e092e710a4cf39fc05 (table test-table [id=a244e0900c43456880ec43d3a8415edb]) successfully deleted
I20260929 11:27:27.415243 32318 ts_tablet_manager.cc:1918] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:27:27.415493 32318 log.cc:1196] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/c4396ef76b3446e092e710a4cf39fc05
I20260929 11:27:27.416215 32318 ts_tablet_manager.cc:1939] T c4396ef76b3446e092e710a4cf39fc05 P 17b9257970fe46b196ae09e375463483: Deleting consensus metadata
I20260929 11:27:27.416885 32294 tablet_bootstrap.cc:492] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483: No bootstrap required, opened a new log
I20260929 11:27:27.417135 32294 ts_tablet_manager.cc:1397] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483: Time spent bootstrapping tablet: real 0.049s user 0.024s sys 0.022s
I20260929 11:27:27.418224 32294 raft_consensus.cc:348] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:27.418358 31767 catalog_manager.cc:4841] TS 17b9257970fe46b196ae09e375463483 (127.23.153.3:44173): tablet c4396ef76b3446e092e710a4cf39fc05 (table test-table [id=a244e0900c43456880ec43d3a8415edb]) successfully deleted
I20260929 11:27:27.418843 32294 raft_consensus.cc:374] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:27.419055 32294 raft_consensus.cc:729] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b9257970fe46b196ae09e375463483, State: Initialized, Role: FOLLOWER
I20260929 11:27:27.419339 32294 consensus_queue.cc:260] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:27.420168 32294 ts_tablet_manager.cc:1428] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 11:27:27.501981 32301 raft_consensus.cc:482] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:27:27.502214 32301 raft_consensus.cc:504] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:27.503916 32301 leader_election.cc:288] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d39a0e45aae94ba9adb9b3bfdb391f36 (127.23.153.1:41057), 17b9257970fe46b196ae09e375463483 (127.23.153.3:44173)
I20260929 11:27:27.511976 32289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:27.505307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:39661 (local address 127.23.153.3:44173)
0929 11:27:27.505490 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:27.505495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:27.505626 (+ 131us) server_negotiation.cc:407] Connection header received
0929 11:27:27.505837 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:27.505841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:27.505899 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:27.505974 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:27.506592 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.507797 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.509999 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.510830 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.510996 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:27.511107 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:27.511494 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 11:27:27.511651 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 11:27:27.512012 32293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:27.504461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:33099 (local address 127.23.153.1:41057)
0929 11:27:27.504666 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:27.504672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:27.504952 (+ 280us) server_negotiation.cc:407] Connection header received
0929 11:27:27.505212 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:27.505218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:27.505298 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:27.505410 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:27.506215 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.507096 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.509470 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.510839 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.511063 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:27.511144 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:27.511613 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 11:27:27.511756 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 11:27:27.512012 32325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:27.505040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:44173 (local address 127.23.153.2:39661)
0929 11:27:27.505541 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:27.505558 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:27.505707 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:27.506010 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:27.506014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:27.506037 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:27.506414 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:27.506426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.507969 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.507988 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:27.509835 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:27.509846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.511088 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.511096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:27.511292 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:27.511328 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 11:27:27.511437 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 11:27:27.511587 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":25,"thread_start_us":449,"threads_started":1}
I20260929 11:27:27.512681 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:27.504291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:41057 (local address 127.23.153.2:33099)
0929 11:27:27.504842 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:27.504862 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:27.505004 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:27.505602 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:27.505608 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:27.505638 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:27.506004 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:27.506017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.507233 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.507238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:27.509252 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:27.509269 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.511272 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.511277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:27.511435 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:27.511461 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 11:27:27.512203 (+ 742us) client_negotiation.cc:240] Negotiation successful
0929 11:27:27.512355 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":17,"thread_start_us":280,"threads_started":1}
I20260929 11:27:27.513378 31939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24b6f5b983054da29f55c1bbdcfaa83f" candidate_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" is_pre_election: true
I20260929 11:27:27.513312 32221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24b6f5b983054da29f55c1bbdcfaa83f" candidate_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b9257970fe46b196ae09e375463483" is_pre_election: true
I20260929 11:27:27.513618 31939 raft_consensus.cc:2454] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66570f7a510a436fa3b2fbbb0d2a907c in term 0.
I20260929 11:27:27.513823 32221 raft_consensus.cc:2454] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66570f7a510a436fa3b2fbbb0d2a907c in term 0.
I20260929 11:27:27.514130 32010 leader_election.cc:302] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66570f7a510a436fa3b2fbbb0d2a907c, d39a0e45aae94ba9adb9b3bfdb391f36; no voters:
I20260929 11:27:27.514562 32301 raft_consensus.cc:2781] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:27:27.514688 32301 raft_consensus.cc:482] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:27:27.514783 32301 raft_consensus.cc:3037] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:27.516976 32301 raft_consensus.cc:504] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:27.517778 32301 leader_election.cc:288] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [CANDIDATE]: Term 1 election: Requested vote from peers d39a0e45aae94ba9adb9b3bfdb391f36 (127.23.153.1:41057), 17b9257970fe46b196ae09e375463483 (127.23.153.3:44173)
I20260929 11:27:27.518493 31939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24b6f5b983054da29f55c1bbdcfaa83f" candidate_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36"
I20260929 11:27:27.518606 32221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24b6f5b983054da29f55c1bbdcfaa83f" candidate_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b9257970fe46b196ae09e375463483"
I20260929 11:27:27.518930 31939 raft_consensus.cc:3037] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:27.518970 32221 raft_consensus.cc:3037] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:27.521852 31939 raft_consensus.cc:2454] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66570f7a510a436fa3b2fbbb0d2a907c in term 1.
I20260929 11:27:27.522658 32010 leader_election.cc:302] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66570f7a510a436fa3b2fbbb0d2a907c, d39a0e45aae94ba9adb9b3bfdb391f36; no voters:
I20260929 11:27:27.523152 32301 raft_consensus.cc:2781] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:27.523409 32221 raft_consensus.cc:2454] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66570f7a510a436fa3b2fbbb0d2a907c in term 1.
I20260929 11:27:27.523911 32301 raft_consensus.cc:686] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 LEADER]: Becoming Leader. State: Replica: 66570f7a510a436fa3b2fbbb0d2a907c, State: Running, Role: LEADER
I20260929 11:27:27.524344 32301 consensus_queue.cc:237] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:27.528389 31780 catalog_manager.cc:5495] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c reported cstate change: term changed from 0 to 1, leader changed from <none> to 66570f7a510a436fa3b2fbbb0d2a907c (127.23.153.2). New cstate: current_term: 1 leader_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } health_report { overall_health: UNKNOWN } } }
Sep 29 11:27:27 dist-test-slave-qk05 krb5kdc[31737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681243, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.62@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM
I20260929 11:27:27.611776 32333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:27.597400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:41277 (local address 127.0.0.1:49752)
0929 11:27:27.598078 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:27.598103 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:27.598298 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:27.598694 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:27.598698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:27.599423 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:27.599677 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:27.599689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.600734 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.600737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:27.601697 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:27.601709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.602384 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.602390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:27.602592 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:27.603395 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:27.603413 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:27.606016 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:27.609206 (+ 3190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:27.609224 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:27.609233 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:27.609681 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:27.610037 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:27.610041 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:27.610045 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:27.610176 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:27.610690 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:27.610699 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:27.610868 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 11:27:27.611043 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 11:27:27.611337 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":355,"threads_started":1}
I20260929 11:27:27.611812 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:27.597474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49752 (local address 127.23.153.62:41277)
0929 11:27:27.598408 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:27.598413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:27.598433 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:27:27.598501 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:27.598505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:27.598573 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:27.598682 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:27.599806 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.600594 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.601826 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.602225 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.602332 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:27.606258 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:27.606263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:27.606266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:27.606299 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:27.609035 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:27.609797 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:27.609802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:27.609806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:27.609929 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:27.610272 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:27.610277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:27.610280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:27.610555 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:27.610694 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:27.611162 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 11:27:27.611474 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":818,"thread_start_us":668,"threads_started":1}
I20260929 11:27:27.617013 31780 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:49752:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 11:27:27.618359 31780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:27:27.626621 31919 tablet_service.cc:1467] Processing CreateTablet for tablet 7d75d3f1732344dda16b729b38474ac1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d37f0e1897c645bb8ad8b92d7f6ce508]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:27:27.627290 31919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d75d3f1732344dda16b729b38474ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:27.629957 32198 tablet_service.cc:1467] Processing CreateTablet for tablet 7d75d3f1732344dda16b729b38474ac1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d37f0e1897c645bb8ad8b92d7f6ce508]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:27:27.630167 32059 tablet_service.cc:1467] Processing CreateTablet for tablet 7d75d3f1732344dda16b729b38474ac1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d37f0e1897c645bb8ad8b92d7f6ce508]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 11:27:27.630524 32198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d75d3f1732344dda16b729b38474ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:27.630752 32059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d75d3f1732344dda16b729b38474ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:27.637351 32294 tablet_bootstrap.cc:492] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483: Bootstrap starting.
I20260929 11:27:27.638379 32295 tablet_bootstrap.cc:492] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c: Bootstrap starting.
I20260929 11:27:27.639164 32294 tablet_bootstrap.cc:654] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:27.640069 32295 tablet_bootstrap.cc:654] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:27.642777 32296 tablet_bootstrap.cc:492] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36: Bootstrap starting.
I20260929 11:27:27.644101 32294 tablet_bootstrap.cc:492] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483: No bootstrap required, opened a new log
I20260929 11:27:27.644373 32294 ts_tablet_manager.cc:1397] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260929 11:27:27.644784 32296 tablet_bootstrap.cc:654] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:27.645109 32294 raft_consensus.cc:348] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:27.645335 32294 raft_consensus.cc:374] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:27.645412 32294 raft_consensus.cc:729] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b9257970fe46b196ae09e375463483, State: Initialized, Role: FOLLOWER
I20260929 11:27:27.646688 32294 consensus_queue.cc:260] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:27.647226 32296 tablet_bootstrap.cc:492] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36: No bootstrap required, opened a new log
I20260929 11:27:27.647397 32296 ts_tablet_manager.cc:1397] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 11:27:27.647298 32300 tablet_replica.cc:439] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } }
I20260929 11:27:27.647505 32294 ts_tablet_manager.cc:1428] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 11:27:27.647598 32300 tablet_replica.cc:442] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:27:27.648161 32296 raft_consensus.cc:348] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:27.648455 32296 raft_consensus.cc:374] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:27.648531 32296 raft_consensus.cc:729] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d39a0e45aae94ba9adb9b3bfdb391f36, State: Initialized, Role: FOLLOWER
I20260929 11:27:27.648794 32296 consensus_queue.cc:260] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:27.649271 32302 tablet_replica.cc:439] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } }
I20260929 11:27:27.649634 32302 tablet_replica.cc:442] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:27:27.650604 32296 ts_tablet_manager.cc:1428] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 11:27:27.652297 32295 tablet_bootstrap.cc:492] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c: No bootstrap required, opened a new log
I20260929 11:27:27.652544 32295 ts_tablet_manager.cc:1397] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.010s
I20260929 11:27:27.653400 32295 raft_consensus.cc:348] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:27.653851 32295 raft_consensus.cc:374] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:27.653980 32295 raft_consensus.cc:729] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66570f7a510a436fa3b2fbbb0d2a907c, State: Initialized, Role: FOLLOWER
I20260929 11:27:27.654273 32295 consensus_queue.cc:260] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:27.654755 32301 tablet_replica.cc:439] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } }
I20260929 11:27:27.654994 32295 ts_tablet_manager.cc:1428] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 11:27:27.655107 32301 tablet_replica.cc:442] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:27:27.733878 32301 raft_consensus.cc:482] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:27:27.734184 32301 raft_consensus.cc:504] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:27.735241 32301 leader_election.cc:288] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d39a0e45aae94ba9adb9b3bfdb391f36 (127.23.153.1:41057), 17b9257970fe46b196ae09e375463483 (127.23.153.3:44173)
I20260929 11:27:27.735849 31939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d75d3f1732344dda16b729b38474ac1" candidate_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" is_pre_election: true
I20260929 11:27:27.735970 32221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d75d3f1732344dda16b729b38474ac1" candidate_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b9257970fe46b196ae09e375463483" is_pre_election: true
I20260929 11:27:27.736137 31939 raft_consensus.cc:2454] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66570f7a510a436fa3b2fbbb0d2a907c in term 0.
I20260929 11:27:27.736253 32221 raft_consensus.cc:2454] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66570f7a510a436fa3b2fbbb0d2a907c in term 0.
I20260929 11:27:27.736887 32010 leader_election.cc:302] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66570f7a510a436fa3b2fbbb0d2a907c, d39a0e45aae94ba9adb9b3bfdb391f36; no voters:
I20260929 11:27:27.737272 32301 raft_consensus.cc:2781] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:27:27.737381 32301 raft_consensus.cc:482] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:27:27.737526 32301 raft_consensus.cc:3037] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:27.739876 32301 raft_consensus.cc:504] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:27.740576 32301 leader_election.cc:288] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [CANDIDATE]: Term 1 election: Requested vote from peers d39a0e45aae94ba9adb9b3bfdb391f36 (127.23.153.1:41057), 17b9257970fe46b196ae09e375463483 (127.23.153.3:44173)
I20260929 11:27:27.741429 32221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d75d3f1732344dda16b729b38474ac1" candidate_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b9257970fe46b196ae09e375463483"
I20260929 11:27:27.741428 31939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d75d3f1732344dda16b729b38474ac1" candidate_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36"
I20260929 11:27:27.741886 32221 raft_consensus.cc:3037] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:27.741958 31939 raft_consensus.cc:3037] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:27.744729 32221 raft_consensus.cc:2454] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66570f7a510a436fa3b2fbbb0d2a907c in term 1.
I20260929 11:27:27.745107 31939 raft_consensus.cc:2454] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66570f7a510a436fa3b2fbbb0d2a907c in term 1.
I20260929 11:27:27.745286 32010 leader_election.cc:302] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b9257970fe46b196ae09e375463483, 66570f7a510a436fa3b2fbbb0d2a907c; no voters:
I20260929 11:27:27.745639 32301 raft_consensus.cc:2781] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:27.745832 32301 raft_consensus.cc:686] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 LEADER]: Becoming Leader. State: Replica: 66570f7a510a436fa3b2fbbb0d2a907c, State: Running, Role: LEADER
I20260929 11:27:27.746132 32301 consensus_queue.cc:237] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:27.748034 32326 tablet_replica.cc:439] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c: TxnStatusTablet state changed. Reason: New leader 66570f7a510a436fa3b2fbbb0d2a907c. Latest consensus state: current_term: 1 leader_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } }
I20260929 11:27:27.748371 32326 tablet_replica.cc:442] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:27:27.750029 31780 catalog_manager.cc:5495] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c reported cstate change: term changed from 0 to 1, leader changed from <none> to 66570f7a510a436fa3b2fbbb0d2a907c (127.23.153.2). New cstate: current_term: 1 leader_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } health_report { overall_health: UNKNOWN } } }
I20260929 11:27:27.758427 32339 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 11:27:27.758719 32339 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 11:27:27.775398 32333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:27.766393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:38847 (local address 127.0.0.1:55434)
0929 11:27:27.766679 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:27.766703 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:27.766885 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:27.768253 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:27.768259 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:27.768275 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:27:27.768488 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:27.768502 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.770172 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.770179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:27.771881 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:27.771904 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.773055 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.773062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:27.773307 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:27.773400 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:27:27.774710 (+ 1310us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:27:27.774740 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 11:27:27.774850 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 11:27:27.775074 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 11:27:27.775441 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:27.766880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55434 (local address 127.23.153.2:38847)
0929 11:27:27.767712 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:27.767719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:27.767746 (+ 27us) server_negotiation.cc:407] Connection header received
0929 11:27:27.767861 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:27.767867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:27.767993 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:27.768173 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:27:27.768659 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.769995 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.772090 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.772840 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.772987 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:27.773707 (+ 720us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:27:27.774556 (+ 849us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:27:27.774719 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:27.774910 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 11:27:27.775053 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":461,"threads_started":1}
I20260929 11:27:27.781903 32221 raft_consensus.cc:1261] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483 [term 1 FOLLOWER]: Refusing update from remote peer 66570f7a510a436fa3b2fbbb0d2a907c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:27:27.782791 31939 raft_consensus.cc:1261] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 FOLLOWER]: Refusing update from remote peer 66570f7a510a436fa3b2fbbb0d2a907c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:27:27.783270 32301 consensus_queue.cc:1035] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:27.783756 32326 consensus_queue.cc:1035] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:27.793260 32300 tablet_replica.cc:439] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483: TxnStatusTablet state changed. Reason: New leader 66570f7a510a436fa3b2fbbb0d2a907c. Latest consensus state: current_term: 1 leader_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } }
I20260929 11:27:27.793684 32300 tablet_replica.cc:442] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:27:27.795791 32302 tablet_replica.cc:439] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36: TxnStatusTablet state changed. Reason: New leader 66570f7a510a436fa3b2fbbb0d2a907c. Latest consensus state: current_term: 1 leader_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } }
I20260929 11:27:27.796141 32302 tablet_replica.cc:442] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:27:27.803113 32310 mvcc.cc:205] Tried to move back new op lower bound from 7334630390903689216 to 7334630390771204096. Current Snapshot: MvccSnapshot[applied={T|T < 7334630390903689216}]
I20260929 11:27:27.804420 32326 tablet_replica.cc:439] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } }
I20260929 11:27:27.804754 32326 tablet_replica.cc:442] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:27:27.806908 32309 mvcc.cc:205] Tried to move back new op lower bound from 7334630390903689216 to 7334630390771204096. Current Snapshot: MvccSnapshot[applied={T|T < 7334630390903689216}]
I20260929 11:27:27.807542 32302 tablet_replica.cc:439] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } }
I20260929 11:27:27.807801 32302 tablet_replica.cc:442] T 7d75d3f1732344dda16b729b38474ac1 P d39a0e45aae94ba9adb9b3bfdb391f36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:27:27.811444 32300 tablet_replica.cc:439] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } }
I20260929 11:27:27.811715 32300 tablet_replica.cc:442] T 7d75d3f1732344dda16b729b38474ac1 P 17b9257970fe46b196ae09e375463483: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 11:27:27.834085 32345 tablet_replica.cc:439] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } } }
I20260929 11:27:27.835688 32345 tablet_replica.cc:442] T 7d75d3f1732344dda16b729b38474ac1 P 66570f7a510a436fa3b2fbbb0d2a907c: This TxnStatusTablet replica's current role is: LEADER
I20260929 11:27:27.843183 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:27.828283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55442 (local address 127.23.153.2:38847)
0929 11:27:27.828515 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:27.828523 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:27.829684 (+ 1161us) server_negotiation.cc:407] Connection header received
0929 11:27:27.830673 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:27.830680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:27.830775 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:27.830845 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:27:27.836961 (+ 6116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.838155 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.840383 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.841163 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.841285 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:27.842125 (+ 840us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:27:27.842277 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:27:27.842345 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:27.842695 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 11:27:27.842795 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:27:27.877281 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:27.869852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:48237 (local address 127.23.153.2:38847)
0929 11:27:27.870068 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:27.870075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:27.870342 (+ 267us) server_negotiation.cc:407] Connection header received
0929 11:27:27.870643 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:27.870649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:27.870739 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:27.870841 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:27:27.871464 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.872698 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.874974 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.875681 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.875816 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:27.876456 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:27:27.876606 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:27:27.876687 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:27.876836 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 11:27:27.876921 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 11:27:27.877616 32351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:27.869392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:38847 (local address 127.23.153.2:48237)
0929 11:27:27.870228 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:27.870250 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:27.870465 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:27.870875 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:27.870880 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:27.870893 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:27:27.871313 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:27.871325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.872916 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.872922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:27.874787 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:27.874806 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:27.875926 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:27.875933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:27.876151 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:27.876231 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:27:27.876710 (+ 479us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:27:27.876721 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 11:27:27.876809 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 11:27:27.876984 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":1,"thread_start_us":660,"threads_started":1}
I20260929 11:27:27.889339 31939 raft_consensus.cc:1261] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 FOLLOWER]: Refusing update from remote peer 66570f7a510a436fa3b2fbbb0d2a907c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:27:27.889580 32221 raft_consensus.cc:1261] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483 [term 1 FOLLOWER]: Refusing update from remote peer 66570f7a510a436fa3b2fbbb0d2a907c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:27:27.890257 32301 consensus_queue.cc:1035] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:27.891135 32345 consensus_queue.cc:1035] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:27.906689 32308 mvcc.cc:205] Tried to move back new op lower bound from 7334630391344615424 to 7334630389863579648. Current Snapshot: MvccSnapshot[applied={T|T < 7334630391344615424}]
I20260929 11:27:27.970976 31780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:27:27.971319 31780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:27:27.974789 31780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39: Sending DeleteTablet for 3 replicas of tablet 24b6f5b983054da29f55c1bbdcfaa83f
I20260929 11:27:27.975987 31919 tablet_service.cc:1514] Processing DeleteTablet for tablet 24b6f5b983054da29f55c1bbdcfaa83f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:27 UTC) from {username='oryx', principal='oryx/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:40194
I20260929 11:27:27.976073 32198 tablet_service.cc:1514] Processing DeleteTablet for tablet 24b6f5b983054da29f55c1bbdcfaa83f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:27 UTC) from {username='oryx', principal='oryx/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:51146
I20260929 11:27:27.978902 32058 tablet_service.cc:1514] Processing DeleteTablet for tablet 24b6f5b983054da29f55c1bbdcfaa83f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:27 UTC) from {username='oryx', principal='oryx/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:48818
I20260929 11:27:28.000402 32357 tablet_replica.cc:330] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c: stopping tablet replica
I20260929 11:27:28.000898 32357 raft_consensus.cc:2229] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:27:28.001811 32357 raft_consensus.cc:2258] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:28.002658 32355 tablet_replica.cc:330] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483: stopping tablet replica
I20260929 11:27:28.004142 32357 ts_tablet_manager.cc:1905] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:28.004671 32355 raft_consensus.cc:2229] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:27:28.005146 32355 raft_consensus.cc:2258] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:28.014309 32355 ts_tablet_manager.cc:1905] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:28.023396 32356 tablet_replica.cc:330] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36: stopping tablet replica
I20260929 11:27:28.023708 32356 raft_consensus.cc:2229] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:27:28.024024 32356 raft_consensus.cc:2258] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:28.025070 32357 ts_tablet_manager.cc:1918] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:27:28.025259 32357 log.cc:1196] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/24b6f5b983054da29f55c1bbdcfaa83f
I20260929 11:27:28.025251 32356 ts_tablet_manager.cc:1905] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:28.025722 32357 ts_tablet_manager.cc:1939] T 24b6f5b983054da29f55c1bbdcfaa83f P 66570f7a510a436fa3b2fbbb0d2a907c: Deleting consensus metadata
I20260929 11:27:28.027041 31767 catalog_manager.cc:4841] TS 66570f7a510a436fa3b2fbbb0d2a907c (127.23.153.2:38847): tablet 24b6f5b983054da29f55c1bbdcfaa83f (table test-table [id=6e97b4b2e79e4325a917ad4626d163dd]) successfully deleted
Sep 29 11:27:28 dist-test-slave-qk05 krb5kdc[31737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260929 11:27:28.031279 32355 ts_tablet_manager.cc:1918] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:27:28.031363 32356 ts_tablet_manager.cc:1918] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:27:28.031476 32356 log.cc:1196] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/24b6f5b983054da29f55c1bbdcfaa83f
I20260929 11:27:28.031661 32355 log.cc:1196] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/24b6f5b983054da29f55c1bbdcfaa83f
I20260929 11:27:28.031919 32356 ts_tablet_manager.cc:1939] T 24b6f5b983054da29f55c1bbdcfaa83f P d39a0e45aae94ba9adb9b3bfdb391f36: Deleting consensus metadata
I20260929 11:27:28.032171 32355 ts_tablet_manager.cc:1939] T 24b6f5b983054da29f55c1bbdcfaa83f P 17b9257970fe46b196ae09e375463483: Deleting consensus metadata
I20260929 11:27:28.033334 31767 catalog_manager.cc:4841] TS d39a0e45aae94ba9adb9b3bfdb391f36 (127.23.153.1:41057): tablet 24b6f5b983054da29f55c1bbdcfaa83f (table test-table [id=6e97b4b2e79e4325a917ad4626d163dd]) successfully deleted
I20260929 11:27:28.033931 31767 catalog_manager.cc:4841] TS 17b9257970fe46b196ae09e375463483 (127.23.153.3:44173): tablet 24b6f5b983054da29f55c1bbdcfaa83f (table test-table [id=6e97b4b2e79e4325a917ad4626d163dd]) successfully deleted
Sep 29 11:27:28 dist-test-slave-qk05 krb5kdc[31737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:28 dist-test-slave-qk05 krb5kdc[31737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 11:27:28.057284 32334 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:28.043633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.23.153.62:41277)
0929 11:27:28.043867 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:28.043873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:28.044214 (+ 341us) server_negotiation.cc:407] Connection header received
0929 11:27:28.044426 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:28.044430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:28.044500 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:28.044580 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:28.046404 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.047342 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.049273 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.049739 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.049849 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:28.056926 (+ 7077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49754: BlockingRecv error: recv got EOF from 127.0.0.1:49754 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 11:27:28 dist-test-slave-qk05 krb5kdc[31737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM
I20260929 11:27:28.079800 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:28.061179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49770 (local address 127.23.153.62:41277)
0929 11:27:28.061338 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:28.061345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:28.061885 (+ 540us) server_negotiation.cc:407] Connection header received
0929 11:27:28.062067 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:28.062072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:28.062142 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:28.062279 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:28.063841 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.064834 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.066424 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.067168 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.067327 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:28.072097 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:28.072105 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:28.072110 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:28.072164 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:28.075981 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:28.077222 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:28.077229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:28.077233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:28.077459 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:28.077992 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:28.077997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:28.078001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:28.078397 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:28.078729 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:28.079126 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 11:27:28.079405 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 11:27:28.084097 31780 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:27:28.084726 31780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:27:28.091806 31919 tablet_service.cc:1467] Processing CreateTablet for tablet 0c91e5336ba14b5cae4147e59a74a54f (DEFAULT_TABLE table=test-table [id=cd273e665384430686fbbf30c05a1117]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:28.091840 32058 tablet_service.cc:1467] Processing CreateTablet for tablet 0c91e5336ba14b5cae4147e59a74a54f (DEFAULT_TABLE table=test-table [id=cd273e665384430686fbbf30c05a1117]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:28.092279 31919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c91e5336ba14b5cae4147e59a74a54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:28.092304 32058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c91e5336ba14b5cae4147e59a74a54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:28.095839 32198 tablet_service.cc:1467] Processing CreateTablet for tablet 0c91e5336ba14b5cae4147e59a74a54f (DEFAULT_TABLE table=test-table [id=cd273e665384430686fbbf30c05a1117]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:28.096333 32198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c91e5336ba14b5cae4147e59a74a54f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:28.099356 32295 tablet_bootstrap.cc:492] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c: Bootstrap starting.
I20260929 11:27:28.101140 32295 tablet_bootstrap.cc:654] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:28.102428 32296 tablet_bootstrap.cc:492] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36: Bootstrap starting.
I20260929 11:27:28.104182 32294 tablet_bootstrap.cc:492] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483: Bootstrap starting.
I20260929 11:27:28.104213 32296 tablet_bootstrap.cc:654] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:28.105906 32294 tablet_bootstrap.cc:654] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:28.107141 32295 tablet_bootstrap.cc:492] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c: No bootstrap required, opened a new log
I20260929 11:27:28.107391 32295 ts_tablet_manager.cc:1397] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 11:27:28.108369 32295 raft_consensus.cc:348] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:28.108675 32295 raft_consensus.cc:374] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:28.108775 32295 raft_consensus.cc:729] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66570f7a510a436fa3b2fbbb0d2a907c, State: Initialized, Role: FOLLOWER
I20260929 11:27:28.109668 32295 consensus_queue.cc:260] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:28.111491 32295 ts_tablet_manager.cc:1428] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 11:27:28.114179 32294 tablet_bootstrap.cc:492] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483: No bootstrap required, opened a new log
I20260929 11:27:28.114450 32294 ts_tablet_manager.cc:1397] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 11:27:28.115404 32294 raft_consensus.cc:348] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:28.115759 32294 raft_consensus.cc:374] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:28.116024 32294 raft_consensus.cc:729] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b9257970fe46b196ae09e375463483, State: Initialized, Role: FOLLOWER
I20260929 11:27:28.116341 32294 consensus_queue.cc:260] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:28.117429 32294 ts_tablet_manager.cc:1428] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 11:27:28.119299 32296 tablet_bootstrap.cc:492] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36: No bootstrap required, opened a new log
I20260929 11:27:28.119572 32296 ts_tablet_manager.cc:1397] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20260929 11:27:28.120638 32296 raft_consensus.cc:348] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:28.120941 32296 raft_consensus.cc:374] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:28.121095 32296 raft_consensus.cc:729] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d39a0e45aae94ba9adb9b3bfdb391f36, State: Initialized, Role: FOLLOWER
I20260929 11:27:28.121392 32296 consensus_queue.cc:260] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:28.123978 32296 ts_tablet_manager.cc:1428] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 11:27:28.358294 32302 raft_consensus.cc:482] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:27:28.358526 32302 raft_consensus.cc:504] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:28.360455 32302 leader_election.cc:288] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66570f7a510a436fa3b2fbbb0d2a907c (127.23.153.2:38847), 17b9257970fe46b196ae09e375463483 (127.23.153.3:44173)
I20260929 11:27:28.371582 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:28.360828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:43509 (local address 127.23.153.2:38847)
0929 11:27:28.361036 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:28.361044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:28.361295 (+ 251us) server_negotiation.cc:407] Connection header received
0929 11:27:28.361573 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:28.361581 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:28.361673 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:28.361740 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:28.362728 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.364165 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.368452 (+ 4287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.370037 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.370217 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:28.370307 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:28.371007 (+ 700us) server_negotiation.cc:299] Negotiation successful
0929 11:27:28.371230 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 11:27:28.372759 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:28.360699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:38847 (local address 127.23.153.1:43509)
0929 11:27:28.361188 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:28.361206 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:28.361348 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:28.362024 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:28.362030 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:28.362063 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:28.362480 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:28.362494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.364817 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.364827 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:28.367981 (+ 3154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:28.368019 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.370497 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.370506 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:28.370762 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:28.370813 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 11:27:28.372123 (+ 1310us) client_negotiation.cc:240] Negotiation successful
0929 11:27:28.372325 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":256,"threads_started":1}
I20260929 11:27:28.374788 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c91e5336ba14b5cae4147e59a74a54f" candidate_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" is_pre_election: true
I20260929 11:27:28.375160 32081 raft_consensus.cc:2454] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d39a0e45aae94ba9adb9b3bfdb391f36 in term 0.
I20260929 11:27:28.375761 31870 leader_election.cc:302] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66570f7a510a436fa3b2fbbb0d2a907c, d39a0e45aae94ba9adb9b3bfdb391f36; no voters:
I20260929 11:27:28.376248 32302 raft_consensus.cc:2781] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:27:28.376387 32302 raft_consensus.cc:482] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:27:28.376492 32302 raft_consensus.cc:3037] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:28.379323 32302 raft_consensus.cc:504] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:28.380103 32302 leader_election.cc:288] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [CANDIDATE]: Term 1 election: Requested vote from peers 66570f7a510a436fa3b2fbbb0d2a907c (127.23.153.2:38847), 17b9257970fe46b196ae09e375463483 (127.23.153.3:44173)
I20260929 11:27:28.380764 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c91e5336ba14b5cae4147e59a74a54f" candidate_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66570f7a510a436fa3b2fbbb0d2a907c"
I20260929 11:27:28.381026 32081 raft_consensus.cc:3037] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:28.381075 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:28.361370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:44173 (local address 127.23.153.1:37591)
0929 11:27:28.373274 (+ 11904us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:28.373299 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:28.373483 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:28.374358 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:28.374364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:28.374399 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:27:28.374612 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:28.374624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.376101 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.376108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:28.378386 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:28.378426 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.380309 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.380316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:28.380475 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:28.380510 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:27:28.380609 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 11:27:28.380788 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11780,"thread_start_us":12637,"threads_started":1}
I20260929 11:27:28.381829 32375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:28.361568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:37591 (local address 127.23.153.3:44173)
0929 11:27:28.374033 (+ 12465us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:28.374040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:28.374060 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:27:28.374135 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:28.374140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:28.374217 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:28.374329 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:27:28.374792 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.375864 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.378644 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.379911 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.380112 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:28.380196 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:28.381302 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0929 11:27:28.381468 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12307,"thread_start_us":12174,"threads_started":1}
I20260929 11:27:28.382659 32221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c91e5336ba14b5cae4147e59a74a54f" candidate_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b9257970fe46b196ae09e375463483" is_pre_election: true
I20260929 11:27:28.382920 32221 raft_consensus.cc:2454] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d39a0e45aae94ba9adb9b3bfdb391f36 in term 0.
I20260929 11:27:28.382906 32220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c91e5336ba14b5cae4147e59a74a54f" candidate_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17b9257970fe46b196ae09e375463483"
I20260929 11:27:28.383188 32220 raft_consensus.cc:3037] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:28.384378 32081 raft_consensus.cc:2454] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d39a0e45aae94ba9adb9b3bfdb391f36 in term 1.
I20260929 11:27:28.384919 31870 leader_election.cc:302] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66570f7a510a436fa3b2fbbb0d2a907c, d39a0e45aae94ba9adb9b3bfdb391f36; no voters:
I20260929 11:27:28.385248 32302 raft_consensus.cc:2781] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:28.385957 32302 raft_consensus.cc:686] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 LEADER]: Becoming Leader. State: Replica: d39a0e45aae94ba9adb9b3bfdb391f36, State: Running, Role: LEADER
I20260929 11:27:28.386431 32220 raft_consensus.cc:2454] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d39a0e45aae94ba9adb9b3bfdb391f36 in term 1.
I20260929 11:27:28.386389 32302 consensus_queue.cc:237] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:28.391176 31781 catalog_manager.cc:5495] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 reported cstate change: term changed from 0 to 1, leader changed from <none> to d39a0e45aae94ba9adb9b3bfdb391f36 (127.23.153.1). New cstate: current_term: 1 leader_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } health_report { overall_health: UNKNOWN } } }
I20260929 11:27:28.427618 32379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:28.419637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39016 (local address 127.23.153.1:41057)
0929 11:27:28.420221 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:28.420225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:28.420247 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:27:28.420341 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:28.420346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:28.420435 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:28.420587 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:27:28.421061 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.421859 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.424157 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.424758 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.424928 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:28.425720 (+ 792us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:27:28.426454 (+ 734us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:27:28.426595 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:28.426972 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 11:27:28.427105 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":344,"threads_started":1}
I20260929 11:27:28.433305 32081 raft_consensus.cc:1261] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 FOLLOWER]: Refusing update from remote peer d39a0e45aae94ba9adb9b3bfdb391f36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:27:28.433334 32220 raft_consensus.cc:1261] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483 [term 1 FOLLOWER]: Refusing update from remote peer d39a0e45aae94ba9adb9b3bfdb391f36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:27:28.434126 32302 consensus_queue.cc:1035] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:28.434531 32376 consensus_queue.cc:1035] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:28.455756 31781 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:27:28.456079 31781 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:27:28.460357 31781 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39: Sending DeleteTablet for 3 replicas of tablet 0c91e5336ba14b5cae4147e59a74a54f
I20260929 11:27:28.461534 32198 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c91e5336ba14b5cae4147e59a74a54f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:28 UTC) from {username='oryx', principal='oryx/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:51146
I20260929 11:27:28.461601 32058 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c91e5336ba14b5cae4147e59a74a54f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:28 UTC) from {username='oryx', principal='oryx/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:48818
I20260929 11:27:28.461874 32355 tablet_replica.cc:330] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483: stopping tablet replica
I20260929 11:27:28.461930 32357 tablet_replica.cc:330] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c: stopping tablet replica
I20260929 11:27:28.462093 32355 raft_consensus.cc:2229] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:27:28.462116 32357 raft_consensus.cc:2229] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:27:28.462339 32357 raft_consensus.cc:2258] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:28.462339 32355 raft_consensus.cc:2258] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:28.463973 32355 ts_tablet_manager.cc:1905] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:28.464082 31919 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c91e5336ba14b5cae4147e59a74a54f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:28 UTC) from {username='oryx', principal='oryx/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:40194
I20260929 11:27:28.464392 32356 tablet_replica.cc:330] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36: stopping tablet replica
I20260929 11:27:28.464591 32356 raft_consensus.cc:2229] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:27:28.465178 32356 raft_consensus.cc:2258] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:28.467247 32357 ts_tablet_manager.cc:1905] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:28.470247 32356 ts_tablet_manager.cc:1905] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:28.472589 32355 ts_tablet_manager.cc:1918] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:27:28.472837 32355 log.cc:1196] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/0c91e5336ba14b5cae4147e59a74a54f
I20260929 11:27:28.473289 32355 ts_tablet_manager.cc:1939] T 0c91e5336ba14b5cae4147e59a74a54f P 17b9257970fe46b196ae09e375463483: Deleting consensus metadata
I20260929 11:27:28.473752 31781 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 11:27:28.474524 31767 catalog_manager.cc:4841] TS 17b9257970fe46b196ae09e375463483 (127.23.153.3:44173): tablet 0c91e5336ba14b5cae4147e59a74a54f (table test-table [id=cd273e665384430686fbbf30c05a1117]) successfully deleted
W20260929 11:27:28.474697 31781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:27:28.480551 32356 ts_tablet_manager.cc:1918] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:27:28.480885 32356 log.cc:1196] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/0c91e5336ba14b5cae4147e59a74a54f
I20260929 11:27:28.481420 32356 ts_tablet_manager.cc:1939] T 0c91e5336ba14b5cae4147e59a74a54f P d39a0e45aae94ba9adb9b3bfdb391f36: Deleting consensus metadata
I20260929 11:27:28.481457 32357 ts_tablet_manager.cc:1918] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:27:28.481638 32357 log.cc:1196] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/0c91e5336ba14b5cae4147e59a74a54f
I20260929 11:27:28.483642 31767 catalog_manager.cc:4841] TS d39a0e45aae94ba9adb9b3bfdb391f36 (127.23.153.1:41057): tablet 0c91e5336ba14b5cae4147e59a74a54f (table test-table [id=cd273e665384430686fbbf30c05a1117]) successfully deleted
I20260929 11:27:28.484066 32357 ts_tablet_manager.cc:1939] T 0c91e5336ba14b5cae4147e59a74a54f P 66570f7a510a436fa3b2fbbb0d2a907c: Deleting consensus metadata
I20260929 11:27:28.487258 31767 catalog_manager.cc:4841] TS 66570f7a510a436fa3b2fbbb0d2a907c (127.23.153.2:38847): tablet 0c91e5336ba14b5cae4147e59a74a54f (table test-table [id=cd273e665384430686fbbf30c05a1117]) successfully deleted
I20260929 11:27:28.488472 32058 tablet_service.cc:1467] Processing CreateTablet for tablet f04f504463fd4f10ac8e17ed9ffc2c0c (DEFAULT_TABLE table=test-table [id=77ad0ad3a968407fbd735f4056389e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:28.488965 32058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f04f504463fd4f10ac8e17ed9ffc2c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:28.488932 32198 tablet_service.cc:1467] Processing CreateTablet for tablet f04f504463fd4f10ac8e17ed9ffc2c0c (DEFAULT_TABLE table=test-table [id=77ad0ad3a968407fbd735f4056389e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:28.489331 32198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f04f504463fd4f10ac8e17ed9ffc2c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:28.495509 31919 tablet_service.cc:1467] Processing CreateTablet for tablet f04f504463fd4f10ac8e17ed9ffc2c0c (DEFAULT_TABLE table=test-table [id=77ad0ad3a968407fbd735f4056389e2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:27:28.496034 31919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f04f504463fd4f10ac8e17ed9ffc2c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:28.497087 32294 tablet_bootstrap.cc:492] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483: Bootstrap starting.
I20260929 11:27:28.498929 32294 tablet_bootstrap.cc:654] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:28.505064 32295 tablet_bootstrap.cc:492] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c: Bootstrap starting.
I20260929 11:27:28.505537 32296 tablet_bootstrap.cc:492] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36: Bootstrap starting.
I20260929 11:27:28.507139 32295 tablet_bootstrap.cc:654] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:28.507346 32296 tablet_bootstrap.cc:654] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:28.516667 32295 tablet_bootstrap.cc:492] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c: No bootstrap required, opened a new log
I20260929 11:27:28.516665 32294 tablet_bootstrap.cc:492] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483: No bootstrap required, opened a new log
I20260929 11:27:28.516865 32294 ts_tablet_manager.cc:1397] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.000s
I20260929 11:27:28.517733 32294 raft_consensus.cc:348] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:28.518029 32294 raft_consensus.cc:374] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:28.517012 32295 ts_tablet_manager.cc:1397] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260929 11:27:28.518103 32294 raft_consensus.cc:729] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b9257970fe46b196ae09e375463483, State: Initialized, Role: FOLLOWER
I20260929 11:27:28.518378 32294 consensus_queue.cc:260] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:28.519140 32295 raft_consensus.cc:348] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:28.519507 32295 raft_consensus.cc:374] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:28.519640 32295 raft_consensus.cc:729] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66570f7a510a436fa3b2fbbb0d2a907c, State: Initialized, Role: FOLLOWER
I20260929 11:27:28.519976 32295 consensus_queue.cc:260] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:28.521881 32294 ts_tablet_manager.cc:1428] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260929 11:27:28.523152 32295 ts_tablet_manager.cc:1428] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 11:27:28.523859 32296 tablet_bootstrap.cc:492] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36: No bootstrap required, opened a new log
I20260929 11:27:28.524047 32296 ts_tablet_manager.cc:1397] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.003s
I20260929 11:27:28.524782 32296 raft_consensus.cc:348] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:28.525060 32296 raft_consensus.cc:374] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:28.525135 32296 raft_consensus.cc:729] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d39a0e45aae94ba9adb9b3bfdb391f36, State: Initialized, Role: FOLLOWER
I20260929 11:27:28.525365 32296 consensus_queue.cc:260] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:28.525883 32296 ts_tablet_manager.cc:1428] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 11:27:28.752458 32300 raft_consensus.cc:482] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:27:28.752692 32300 raft_consensus.cc:504] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:28.753443 32300 leader_election.cc:288] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d39a0e45aae94ba9adb9b3bfdb391f36 (127.23.153.1:41057), 66570f7a510a436fa3b2fbbb0d2a907c (127.23.153.2:38847)
I20260929 11:27:28.754048 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04f504463fd4f10ac8e17ed9ffc2c0c" candidate_uuid: "17b9257970fe46b196ae09e375463483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" is_pre_election: true
I20260929 11:27:28.754228 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04f504463fd4f10ac8e17ed9ffc2c0c" candidate_uuid: "17b9257970fe46b196ae09e375463483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" is_pre_election: true
I20260929 11:27:28.754379 31938 raft_consensus.cc:2454] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17b9257970fe46b196ae09e375463483 in term 0.
I20260929 11:27:28.754490 32081 raft_consensus.cc:2454] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17b9257970fe46b196ae09e375463483 in term 0.
I20260929 11:27:28.754920 32150 leader_election.cc:302] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b9257970fe46b196ae09e375463483, d39a0e45aae94ba9adb9b3bfdb391f36; no voters:
I20260929 11:27:28.755286 32300 raft_consensus.cc:2781] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:27:28.755421 32300 raft_consensus.cc:482] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:27:28.755514 32300 raft_consensus.cc:3037] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:28.758872 32300 raft_consensus.cc:504] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:28.759831 32300 leader_election.cc:288] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [CANDIDATE]: Term 1 election: Requested vote from peers d39a0e45aae94ba9adb9b3bfdb391f36 (127.23.153.1:41057), 66570f7a510a436fa3b2fbbb0d2a907c (127.23.153.2:38847)
I20260929 11:27:28.760540 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04f504463fd4f10ac8e17ed9ffc2c0c" candidate_uuid: "17b9257970fe46b196ae09e375463483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66570f7a510a436fa3b2fbbb0d2a907c"
I20260929 11:27:28.760883 32081 raft_consensus.cc:3037] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:28.765324 32081 raft_consensus.cc:2454] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17b9257970fe46b196ae09e375463483 in term 1.
I20260929 11:27:28.770635 31938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04f504463fd4f10ac8e17ed9ffc2c0c" candidate_uuid: "17b9257970fe46b196ae09e375463483" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36"
I20260929 11:27:28.770915 31938 raft_consensus.cc:3037] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:28.773797 31938 raft_consensus.cc:2454] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17b9257970fe46b196ae09e375463483 in term 1.
I20260929 11:27:28.774438 32150 leader_election.cc:302] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17b9257970fe46b196ae09e375463483, d39a0e45aae94ba9adb9b3bfdb391f36; no voters:
I20260929 11:27:28.774772 32300 raft_consensus.cc:2781] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:28.775559 32300 raft_consensus.cc:686] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [term 1 LEADER]: Becoming Leader. State: Replica: 17b9257970fe46b196ae09e375463483, State: Running, Role: LEADER
I20260929 11:27:28.776111 32300 consensus_queue.cc:237] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } }
I20260929 11:27:28.779001 31780 catalog_manager.cc:5495] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17b9257970fe46b196ae09e375463483 (127.23.153.3). New cstate: current_term: 1 leader_uuid: "17b9257970fe46b196ae09e375463483" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17b9257970fe46b196ae09e375463483" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44173 } health_report { overall_health: HEALTHY } } }
I20260929 11:27:28.845266 32375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:28.837567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54112 (local address 127.23.153.3:44173)
0929 11:27:28.838039 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:28.838046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:28.838067 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:27:28.838145 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:28.838151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:28.838245 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:28.838333 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:27:28.838915 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.839993 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.841813 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.842543 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.842677 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:28.843615 (+ 938us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:27:28.843845 (+ 230us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:27:28.843959 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:28.844821 (+ 862us) server_negotiation.cc:299] Negotiation successful
0929 11:27:28.844941 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326}
I20260929 11:27:28.862968 32393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:28.856942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:38847 (local address 127.23.153.3:55383)
0929 11:27:28.857740 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:28.857756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:28.857933 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:28.858277 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:28.858281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:28.858293 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:27:28.858561 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:28.858571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.859719 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.859725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:28.860948 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:28.860962 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.861743 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.861750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:28.861910 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:28.861981 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:27:28.862357 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:27:28.862368 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 11:27:28.862443 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 11:27:28.862588 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":399,"threads_started":1}
I20260929 11:27:28.862957 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:28.857044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:55383 (local address 127.23.153.2:38847)
0929 11:27:28.857234 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:28.857241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:28.857859 (+ 618us) server_negotiation.cc:407] Connection header received
0929 11:27:28.858085 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:28.858090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:28.858167 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:28.858244 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:27:28.858774 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.859584 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.861097 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.861558 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.861671 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:28.862148 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:27:28.862266 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:27:28.862337 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:28.862471 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 11:27:28.862563 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:27:28.873003 32081 raft_consensus.cc:1261] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 FOLLOWER]: Refusing update from remote peer 17b9257970fe46b196ae09e375463483: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:27:28.873078 31938 raft_consensus.cc:1261] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 FOLLOWER]: Refusing update from remote peer 17b9257970fe46b196ae09e375463483: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:27:28.873625 32300 consensus_queue.cc:1035] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66570f7a510a436fa3b2fbbb0d2a907c" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 38847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:28.873945 32386 consensus_queue.cc:1035] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d39a0e45aae94ba9adb9b3bfdb391f36" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 41057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:27:28.920265 32375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:28.911985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:55831 (local address 127.23.153.3:44173)
0929 11:27:28.912153 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:28.912158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:28.912404 (+ 246us) server_negotiation.cc:407] Connection header received
0929 11:27:28.912759 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:28.912765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:28.912866 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:28.912943 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:27:28.915059 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.915922 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.918134 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.918682 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.918820 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:28.919445 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:27:28.919594 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:27:28.919678 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:28.919832 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 11:27:28.919924 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:27:28.920288 32399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:28.911598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:44173 (local address 127.23.153.2:55831)
0929 11:27:28.912283 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:28.912308 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:28.912565 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:28.914553 (+ 1988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:28.914560 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:28.914578 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 11:27:28.914847 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:28.914862 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.916095 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.916101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:28.917726 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:28.917752 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:28.918903 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:28.918910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:28.919142 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:28.919200 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 11:27:28.919698 (+ 498us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 11:27:28.919711 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 11:27:28.919791 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 11:27:28.919946 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":653,"threads_started":1}
I20260929 11:27:28.968950 31781 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:27:28.969226 31781 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:27:28.972400 31781 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85f4546e728a4a1cbe83fa0d789f5c39: Sending DeleteTablet for 3 replicas of tablet f04f504463fd4f10ac8e17ed9ffc2c0c
I20260929 11:27:28.973536 32058 tablet_service.cc:1514] Processing DeleteTablet for tablet f04f504463fd4f10ac8e17ed9ffc2c0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:28 UTC) from {username='oryx', principal='oryx/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:48818
I20260929 11:27:28.973673 31919 tablet_service.cc:1514] Processing DeleteTablet for tablet f04f504463fd4f10ac8e17ed9ffc2c0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:28 UTC) from {username='oryx', principal='oryx/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:40194
I20260929 11:27:28.973680 32198 tablet_service.cc:1514] Processing DeleteTablet for tablet f04f504463fd4f10ac8e17ed9ffc2c0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:27:28 UTC) from {username='oryx', principal='oryx/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:51146
I20260929 11:27:28.973954 32357 tablet_replica.cc:330] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c: stopping tablet replica
I20260929 11:27:28.974140 32357 raft_consensus.cc:2229] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:27:28.974349 32357 raft_consensus.cc:2258] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:28.974700 32356 tablet_replica.cc:330] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36: stopping tablet replica
I20260929 11:27:28.974709 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 31853
I20260929 11:27:28.974895 32356 raft_consensus.cc:2229] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:27:28.975145 32356 raft_consensus.cc:2258] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:28.975462 32355 tablet_replica.cc:330] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483: stopping tablet replica
I20260929 11:27:28.975649 32355 raft_consensus.cc:2229] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:27:28.975997 32355 raft_consensus.cc:2258] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:27:28.977200 32355 ts_tablet_manager.cc:1905] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:28.978448 32356 ts_tablet_manager.cc:1905] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:28.984246 32355 ts_tablet_manager.cc:1918] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:27:28.984337 32357 ts_tablet_manager.cc:1905] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:27:28.984745 32355 log.cc:1196] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/f04f504463fd4f10ac8e17ed9ffc2c0c
I20260929 11:27:28.985272 32355 ts_tablet_manager.cc:1939] T f04f504463fd4f10ac8e17ed9ffc2c0c P 17b9257970fe46b196ae09e375463483: Deleting consensus metadata
I20260929 11:27:28.986778 31767 catalog_manager.cc:4841] TS 17b9257970fe46b196ae09e375463483 (127.23.153.3:44173): tablet f04f504463fd4f10ac8e17ed9ffc2c0c (table test-table [id=77ad0ad3a968407fbd735f4056389e2e]) successfully deleted
I20260929 11:27:28.992198 32357 ts_tablet_manager.cc:1918] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:27:28.992412 32357 log.cc:1196] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/f04f504463fd4f10ac8e17ed9ffc2c0c
I20260929 11:27:28.992866 32357 ts_tablet_manager.cc:1939] T f04f504463fd4f10ac8e17ed9ffc2c0c P 66570f7a510a436fa3b2fbbb0d2a907c: Deleting consensus metadata
I20260929 11:27:28.994355 31767 catalog_manager.cc:4841] TS 66570f7a510a436fa3b2fbbb0d2a907c (127.23.153.2:38847): tablet f04f504463fd4f10ac8e17ed9ffc2c0c (table test-table [id=77ad0ad3a968407fbd735f4056389e2e]) successfully deleted
I20260929 11:27:28.997996 32356 ts_tablet_manager.cc:1918] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 11:27:28.998180 32356 log.cc:1196] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/f04f504463fd4f10ac8e17ed9ffc2c0c
I20260929 11:27:28.998646 32356 ts_tablet_manager.cc:1939] T f04f504463fd4f10ac8e17ed9ffc2c0c P d39a0e45aae94ba9adb9b3bfdb391f36: Deleting consensus metadata
I20260929 11:27:28.999867 31767 catalog_manager.cc:4841] TS d39a0e45aae94ba9adb9b3bfdb391f36 (127.23.153.1:41057): tablet f04f504463fd4f10ac8e17ed9ffc2c0c (table test-table [id=77ad0ad3a968407fbd735f4056389e2e]) successfully deleted
Sep 29 11:27:29 dist-test-slave-qk05 krb5kdc[31737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.153.1@KRBTEST.COM
I20260929 11:27:29.015151 32402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:28.976099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.23.153.1:41057)
0929 11:27:28.991556 (+ 15457us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:28.991563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:28.998623 (+ 7060us) server_negotiation.cc:407] Connection header received
0929 11:27:28.998910 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:28.998916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:28.998997 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:28.999150 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:29.000849 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:29.001822 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:29.003665 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:29.004131 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:29.004250 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:29.008756 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:29.008772 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:29.008789 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:29.008835 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:29.012003 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:29.012825 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:29.012834 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:29.012836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:29.013007 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:29.013378 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:29.013382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:29.013384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:29.013768 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:29.014051 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:29.014396 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 11:27:29.014629 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15299,"thread_start_us":15167,"threads_started":1}
I20260929 11:27:29.015918 31984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:29.148293 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 31853
I20260929 11:27:29.171579 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 31993
Sep 29 11:27:29 dist-test-slave-qk05 krb5kdc[31737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.153.2@KRBTEST.COM
I20260929 11:27:29.190404 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:29.172869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55450 (local address 127.23.153.2:38847)
0929 11:27:29.173076 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:29.173081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:29.173100 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:27:29.173156 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:29.173160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:29.173250 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:29.173373 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:29.174799 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:29.175584 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:29.177051 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:29.177471 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:29.177573 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:29.182222 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:29.182241 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:29.182258 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:29.182302 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:29.186679 (+ 4377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:29.187587 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:29.187596 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:29.187598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:29.187750 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:29.188174 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:29.188178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:29.188180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:29.188850 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:29.189263 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:29.189602 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 11:27:29.189858 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 11:27:29.191259 32124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:29.340237 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 31993
I20260929 11:27:29.364504 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 32133
Sep 29 11:27:29 dist-test-slave-qk05 krb5kdc[31737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.153.3@KRBTEST.COM
I20260929 11:27:29.380812 32375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:29.365419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.23.153.3:44173)
0929 11:27:29.365620 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:29.365628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:29.365654 (+ 26us) server_negotiation.cc:407] Connection header received
0929 11:27:29.365727 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:29.365733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:29.365799 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:29.365896 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:29.367351 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:29.368057 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:29.369699 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:29.370118 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:29.370231 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:29.374925 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:29.374942 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:29.374960 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:29.375007 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:29.378132 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:29.378892 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:29.378900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:29.378902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:29.379083 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:29.379456 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:29.379459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:29.379461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:29.379842 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:29.380071 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:29.380288 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 11:27:29.380493 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 11:27:29.381485 32264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:29.508649 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 32133
I20260929 11:27:29.527194 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 31750
I20260929 11:27:29.528733 31822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:29.644600 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 31750
2026-09-29T11:27:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7388 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:27:29 dist-test-slave-qk05 krb5kdc[32409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:27:29 dist-test-slave-qk05 krb5kdc[32409](info): set up 2 sockets
Sep 29 11:27:29 dist-test-slave-qk05 krb5kdc[32409](info): commencing operation
krb5kdc: starting...
W20260929 11:27:30.683980 24164 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:27:30 dist-test-slave-qk05 krb5kdc[32409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:27:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:27:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:27:30.910960 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:35319
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:35319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:31.253324 32422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:31.253661 32422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:31.253785 32422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:31.261982 32422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:27:31.262115 32422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:31.262187 32422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:31.262264 32422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:27:31.262326 32422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:27:31.272933 32422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:35319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:35319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:31.274498 32422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:31.275939 32422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:31.288591 32427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:31.288524 32428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:31.288627 32430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:31.288945 32422 server_base.cc:1034] running on GCE node
I20260929 11:27:31.290122 32422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:31.291724 32422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:31.292959 32422 hybrid_clock.cc:648] HybridClock initialized: now 1790681251292897 us; error 51 us; skew 500 ppm
I20260929 11:27:31.293327 32422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T11:27:31Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1718 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:27:31 dist-test-slave-qk05 krb5kdc[32436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:27:31 dist-test-slave-qk05 krb5kdc[32436](info): set up 2 sockets
Sep 29 11:27:31 dist-test-slave-qk05 krb5kdc[32436](info): commencing operation
krb5kdc: starting...
W20260929 11:27:32.399755 24164 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.004s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:27:32 dist-test-slave-qk05 krb5kdc[32436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:27:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:27:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:27:32.639564 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:42209
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:44157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:42209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:32.990608 32449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:32.990935 32449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:32.991082 32449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:32.998615 32449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:27:32.998749 32449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:32.998793 32449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:32.998864 32449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:27:32.998926 32449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:27:33.010275 32449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:44157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:42209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:42209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:33.011770 32449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:33.013123 32449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:33.025934 32454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:33.025918 32455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:33.026304 32457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:33.028012 32449 server_base.cc:1034] running on GCE node
I20260929 11:27:33.029075 32449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:33.030799 32449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:33.032083 32449 hybrid_clock.cc:648] HybridClock initialized: now 1790681253031998 us; error 78 us; skew 500 ppm
I20260929 11:27:33.032547 32449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:33 dist-test-slave-qk05 krb5kdc[32436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681253, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:33.040462 32449 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:27:33.042217 32449 webserver.cc:460] Webserver started at http://127.23.153.62:39877/ using document root <none> and password file <none>
I20260929 11:27:33.042985 32449 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:33.043172 32449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:33.043500 32449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:33.046614 32449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "eb0449cb93c243b79fa0ffdcdc9fed42"
format_stamp: "Formatted at 2026-09-29 11:27:33 on dist-test-slave-qk05"
server_key: "f20c103ceda12ffd9ff27178a96ac0be"
server_key_iv: "a157234f7e2b45e836a6f6bbcf30575d"
server_key_version: "encryptionkey@0"
I20260929 11:27:33.047449 32449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "eb0449cb93c243b79fa0ffdcdc9fed42"
format_stamp: "Formatted at 2026-09-29 11:27:33 on dist-test-slave-qk05"
server_key: "f20c103ceda12ffd9ff27178a96ac0be"
server_key_iv: "a157234f7e2b45e836a6f6bbcf30575d"
server_key_version: "encryptionkey@0"
I20260929 11:27:33.053186 32449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 11:27:33.057276 32464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:33.058732 32449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:27:33.058913 32449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "eb0449cb93c243b79fa0ffdcdc9fed42"
format_stamp: "Formatted at 2026-09-29 11:27:33 on dist-test-slave-qk05"
server_key: "f20c103ceda12ffd9ff27178a96ac0be"
server_key_iv: "a157234f7e2b45e836a6f6bbcf30575d"
server_key_version: "encryptionkey@0"
I20260929 11:27:33.059173 32449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:33.080597 32449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:33.085290 32449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:33.085584 32449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:33.104351 32449 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:42209
I20260929 11:27:33.104382 32515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:42209 every 8 connection(s)
I20260929 11:27:33.106148 32449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:27:33.110173 32516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:33.112941 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 32449
I20260929 11:27:33.113164 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:27:33.113929 24164 external_mini_cluster.cc:1428] Setting key d8263a16c78b05d7b5d85b528340ea94
I20260929 11:27:33.124223 32516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42: Bootstrap starting.
I20260929 11:27:33.127885 32516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:33.129046 32516 log.cc:824] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42: Log is configured to *not* fsync() on all Append() calls
Sep 29 11:27:33 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:27:33.132686 32516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42: No bootstrap required, opened a new log
I20260929 11:27:33.139925 32519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:33.116254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.23.153.62:42209)
0929 11:27:33.117161 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:33.117174 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:33.117222 (+ 48us) server_negotiation.cc:407] Connection header received
0929 11:27:33.119248 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:33.119278 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:33.119780 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:33.120184 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:33.122663 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:33.124276 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:33.125692 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:33.126478 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:33.126689 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:33.131054 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:33.131066 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:33.131090 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:33.131134 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:33.134456 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:33.135630 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:33.135645 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:33.135656 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:33.135876 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:33.136345 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:33.136360 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:33.136362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:33.137009 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:33.137266 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:33.138180 (+ 914us) server_negotiation.cc:299] Negotiation successful
0929 11:27:33.138678 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":426,"threads_started":1}
I20260929 11:27:33.142763 32516 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0449cb93c243b79fa0ffdcdc9fed42" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42209 } }
I20260929 11:27:33.143342 32516 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:33.143474 32516 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0449cb93c243b79fa0ffdcdc9fed42, State: Initialized, Role: FOLLOWER
I20260929 11:27:33.144047 32516 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0449cb93c243b79fa0ffdcdc9fed42" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42209 } }
I20260929 11:27:33.144289 32516 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:27:33.144416 32516 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:27:33.144588 32516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:33.146888 32516 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0449cb93c243b79fa0ffdcdc9fed42" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42209 } }
I20260929 11:27:33.147534 32516 leader_election.cc:302] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb0449cb93c243b79fa0ffdcdc9fed42; no voters:
I20260929 11:27:33.148203 32516 leader_election.cc:288] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:27:33.148370 32521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:33.149356 32521 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [term 1 LEADER]: Becoming Leader. State: Replica: eb0449cb93c243b79fa0ffdcdc9fed42, State: Running, Role: LEADER
I20260929 11:27:33.149976 32521 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0449cb93c243b79fa0ffdcdc9fed42" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42209 } }
I20260929 11:27:33.150739 32516 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:27:33.153064 32522 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb0449cb93c243b79fa0ffdcdc9fed42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0449cb93c243b79fa0ffdcdc9fed42" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42209 } } }
I20260929 11:27:33.153501 32522 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:33.153967 32523 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb0449cb93c243b79fa0ffdcdc9fed42. Latest consensus state: current_term: 1 leader_uuid: "eb0449cb93c243b79fa0ffdcdc9fed42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0449cb93c243b79fa0ffdcdc9fed42" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42209 } } }
I20260929 11:27:33.155054 32523 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:33.155616 32530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:27:33.162497 32530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:27:33.173822 32530 catalog_manager.cc:1347] Generated new cluster ID: a34ecc6b87314beb9e5236a2ab8241b2
I20260929 11:27:33.173957 32530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:27:33.200896 32530 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:27:33.203066 32530 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:27:33.212095 32530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42: Generated new TSK 0
I20260929 11:27:33.213277 32530 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
I20260929 11:27:33.268060 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42209
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:44157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:33.640225 32542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:33.640547 32542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:33.640689 32542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:33.648934 32542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:33.649215 32542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:33.649441 32542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:27:33.660769 32542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:44157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:33.662081 32542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:33.663584 32542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:33.677310 32548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:33.677345 32550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:33.677369 32547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:33.678216 32542 server_base.cc:1034] running on GCE node
I20260929 11:27:33.678867 32542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:33.679952 32542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:33.681202 32542 hybrid_clock.cc:648] HybridClock initialized: now 1790681253681142 us; error 47 us; skew 500 ppm
I20260929 11:27:33.681566 32542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:33 dist-test-slave-qk05 krb5kdc[32436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681253, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:33.685213 32542 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:27:33.687784 32542 webserver.cc:460] Webserver started at http://127.23.153.1:39271/ using document root <none> and password file <none>
I20260929 11:27:33.688709 32542 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:33.688853 32542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:33.689201 32542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:33.692466 32542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "837f05ddce394df1b278e53ed317d6c1"
format_stamp: "Formatted at 2026-09-29 11:27:33 on dist-test-slave-qk05"
server_key: "e81e982ab60cb5054a7212205b395570"
server_key_iv: "1808ea48ab2129f65e4ad90b950ef546"
server_key_version: "encryptionkey@0"
I20260929 11:27:33.693322 32542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "837f05ddce394df1b278e53ed317d6c1"
format_stamp: "Formatted at 2026-09-29 11:27:33 on dist-test-slave-qk05"
server_key: "e81e982ab60cb5054a7212205b395570"
server_key_iv: "1808ea48ab2129f65e4ad90b950ef546"
server_key_version: "encryptionkey@0"
I20260929 11:27:33.700021 32542 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 11:27:33.704962 32557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:33.706936 32542 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:27:33.707435 32542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "837f05ddce394df1b278e53ed317d6c1"
format_stamp: "Formatted at 2026-09-29 11:27:33 on dist-test-slave-qk05"
server_key: "e81e982ab60cb5054a7212205b395570"
server_key_iv: "1808ea48ab2129f65e4ad90b950ef546"
server_key_version: "encryptionkey@0"
I20260929 11:27:33.707692 32542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:33.737190 32542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:33.742210 32542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:33.742523 32542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:33.743635 32542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:33.745060 32542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:33.745146 32542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:33.745273 32542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:33.745339 32542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:33.778013 32542 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:39593
I20260929 11:27:33.778061 32669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:39593 every 8 connection(s)
I20260929 11:27:33.779428 32542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:27:33.782287 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 32542
I20260929 11:27:33.782483 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:27:33.783136 24164 external_mini_cluster.cc:1428] Setting key c234b2009c269f2f6058380a71137f5a
Sep 29 11:27:33 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681253, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:27:33.802018 32672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:33.781931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:33441 (local address 127.23.153.62:42209)
0929 11:27:33.782417 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:33.782424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:33.783596 (+ 1172us) server_negotiation.cc:407] Connection header received
0929 11:27:33.785769 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:33.785774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:33.785867 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:33.785984 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:33.787620 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:33.788568 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:33.790280 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:33.790858 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:33.791001 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:33.795398 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:33.795407 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:33.795412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:33.795461 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:33.798515 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:33.799502 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:33.799508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:33.799511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:33.799668 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:33.800063 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:33.800067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:33.800070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:33.800447 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:33.800599 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:33.801375 (+ 776us) server_negotiation.cc:299] Negotiation successful
0929 11:27:33.801667 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":237,"threads_started":1}
I20260929 11:27:33.803058 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:33.782506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.1:33441)
0929 11:27:33.783386 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:33.783440 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:33.785476 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:33.786198 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:33.786207 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:33.786760 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:33.787419 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:33.787440 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:33.788735 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:33.788740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:33.790057 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:33.790072 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:33.791119 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:33.791125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:33.791347 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:33.792064 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:33.792104 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:33.794911 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:33.798741 (+ 3830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:33.798750 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:33.798768 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:33.799353 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:33.799769 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:33.799772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:33.799775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:33.799961 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:33.800571 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:33.800579 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:33.800950 (+ 371us) client_negotiation.cc:769] Sending connection context
0929 11:27:33.801218 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 11:27:33.801774 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":351,"threads_started":1}
I20260929 11:27:33.805253 32670 heartbeater.cc:347] Connected to a master server at 127.23.153.62:42209
I20260929 11:27:33.805595 32670 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.153.2@KRBTEST.COM; defaulting to no policy
I20260929 11:27:33.806416 32670 heartbeater.cc:511] Master 127.23.153.62:42209 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.2@KRBTEST.COM" created.
I20260929 11:27:33.809355 32481 ts_manager.cc:194] Registered new tserver with Master: 837f05ddce394df1b278e53ed317d6c1 (127.23.153.1:39593)
I20260929 11:27:33.810911 32481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:33441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
I20260929 11:27:33.835431 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42209
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:44157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:34.209189 32676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:34.209566 32676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:34.209733 32676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:34.219146 32676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:34.219290 32676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:34.219444 32676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:27:34.230504 32676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:44157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:34.232245 32676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:34.233844 32676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:34.247494 32684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:34.247515 32682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:34.247628 32681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:34.248410 32676 server_base.cc:1034] running on GCE node
I20260929 11:27:34.249092 32676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:34.250191 32676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:34.251402 32676 hybrid_clock.cc:648] HybridClock initialized: now 1790681254251350 us; error 52 us; skew 500 ppm
I20260929 11:27:34.251780 32676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:34 dist-test-slave-qk05 krb5kdc[32436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681254, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:34.255839 32676 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:27:34.257594 32676 webserver.cc:460] Webserver started at http://127.23.153.2:35543/ using document root <none> and password file <none>
I20260929 11:27:34.258549 32676 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:34.258705 32676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:34.259078 32676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:34.262422 32676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "384bd752a12e4532867ee58b835e8f5f"
format_stamp: "Formatted at 2026-09-29 11:27:34 on dist-test-slave-qk05"
server_key: "1f060ca9db72aa3a01917d43252b2406"
server_key_iv: "202a544aed4d628b5ef47955bf2d4116"
server_key_version: "encryptionkey@0"
I20260929 11:27:34.263684 32676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "384bd752a12e4532867ee58b835e8f5f"
format_stamp: "Formatted at 2026-09-29 11:27:34 on dist-test-slave-qk05"
server_key: "1f060ca9db72aa3a01917d43252b2406"
server_key_iv: "202a544aed4d628b5ef47955bf2d4116"
server_key_version: "encryptionkey@0"
I20260929 11:27:34.270843 32676 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260929 11:27:34.275377 32691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:34.277086 32676 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:27:34.277299 32676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "384bd752a12e4532867ee58b835e8f5f"
format_stamp: "Formatted at 2026-09-29 11:27:34 on dist-test-slave-qk05"
server_key: "1f060ca9db72aa3a01917d43252b2406"
server_key_iv: "202a544aed4d628b5ef47955bf2d4116"
server_key_version: "encryptionkey@0"
I20260929 11:27:34.277529 32676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:34.298051 32676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:34.302731 32676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:34.303069 32676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:34.304442 32676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:34.306358 32676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:34.306463 32676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:34.306608 32676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:34.306658 32676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:34.341229 32676 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:43843
I20260929 11:27:34.341272 335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:43843 every 8 connection(s)
I20260929 11:27:34.342618 32676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:27:34.345618 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 32676
I20260929 11:27:34.345881 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:27:34.346402 24164 external_mini_cluster.cc:1428] Setting key 352c2683f15880102bbb57690f010e2c
Sep 29 11:27:34 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681254, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:27:34.368961 338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:34.346254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:46357 (local address 127.23.153.62:42209)
0929 11:27:34.346785 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:34.346792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:34.348467 (+ 1675us) server_negotiation.cc:407] Connection header received
0929 11:27:34.350877 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:34.350882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:34.350989 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:34.351136 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:34.353490 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:34.354338 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:34.356847 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:34.357539 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:34.357691 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:34.361624 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:34.361631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:34.361636 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:34.361679 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:34.365169 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:34.366192 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:34.366198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:34.366202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:34.366372 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:34.366863 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:34.366868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:34.366871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:34.367293 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:34.367466 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:34.368302 (+ 836us) server_negotiation.cc:299] Negotiation successful
0929 11:27:34.368571 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":274,"threads_started":1}
WARNING: no policy specified for kudu/127.23.153.3@KRBTEST.COM; defaulting to no policy
I20260929 11:27:34.369854 339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:34.347177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.2:46357)
0929 11:27:34.348225 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:34.348279 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:34.350531 (+ 2252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:34.351487 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:34.351500 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:34.352328 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:34.353211 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:34.353240 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:34.354538 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:34.354544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:34.356606 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:34.356628 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:34.357811 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:34.357816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:34.358152 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:34.358912 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:34.358945 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:34.361294 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:34.365370 (+ 4076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:34.365378 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:34.365397 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:34.365936 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:34.366516 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:34.366519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:34.366522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:34.366734 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:34.367440 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:34.367448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:34.367835 (+ 387us) client_negotiation.cc:769] Sending connection context
0929 11:27:34.368177 (+ 342us) client_negotiation.cc:240] Negotiation successful
0929 11:27:34.368645 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":364,"threads_started":1}
I20260929 11:27:34.371480 336 heartbeater.cc:347] Connected to a master server at 127.23.153.62:42209
I20260929 11:27:34.371793 336 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.3@KRBTEST.COM" created.
I20260929 11:27:34.372736 336 heartbeater.cc:511] Master 127.23.153.62:42209 requested a full tablet report, sending...
I20260929 11:27:34.374459 32481 ts_manager.cc:194] Registered new tserver with Master: 384bd752a12e4532867ee58b835e8f5f (127.23.153.2:43843)
I20260929 11:27:34.375525 32481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:46357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
I20260929 11:27:34.398909 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42209
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:44157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:34.754586 342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:34.754895 342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:34.755059 342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:34.762562 342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:34.762707 342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:34.762877 342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:27:34.773801 342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:44157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:34.775302 342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:34.776679 342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:34.792137 347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:34.792155 348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:34.792213 350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:34.793040 342 server_base.cc:1034] running on GCE node
I20260929 11:27:34.793697 342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:34.794771 342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:34.796011 342 hybrid_clock.cc:648] HybridClock initialized: now 1790681254795955 us; error 44 us; skew 500 ppm
I20260929 11:27:34.796373 342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:34 dist-test-slave-qk05 krb5kdc[32436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681254, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:34.799873 342 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:27:34.801450 342 webserver.cc:460] Webserver started at http://127.23.153.3:46277/ using document root <none> and password file <none>
I20260929 11:27:34.802294 342 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:34.802438 342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:34.802791 342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:34.805967 342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "48b959d3da724449800f64a1aeef9446"
format_stamp: "Formatted at 2026-09-29 11:27:34 on dist-test-slave-qk05"
server_key: "eda5316af0d62f4f3cbbb74254a5e8de"
server_key_iv: "0ec36d8902c09490859853de3e0e7993"
server_key_version: "encryptionkey@0"
I20260929 11:27:34.806836 342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "48b959d3da724449800f64a1aeef9446"
format_stamp: "Formatted at 2026-09-29 11:27:34 on dist-test-slave-qk05"
server_key: "eda5316af0d62f4f3cbbb74254a5e8de"
server_key_iv: "0ec36d8902c09490859853de3e0e7993"
server_key_version: "encryptionkey@0"
I20260929 11:27:34.812733 342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 11:27:34.814684 32670 heartbeater.cc:503] Master 127.23.153.62:42209 was elected leader, sending a full tablet report...
I20260929 11:27:34.817018 357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:34.818519 342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 11:27:34.818707 342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "48b959d3da724449800f64a1aeef9446"
format_stamp: "Formatted at 2026-09-29 11:27:34 on dist-test-slave-qk05"
server_key: "eda5316af0d62f4f3cbbb74254a5e8de"
server_key_iv: "0ec36d8902c09490859853de3e0e7993"
server_key_version: "encryptionkey@0"
I20260929 11:27:34.818912 342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:34.837885 342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:34.842574 342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:34.842859 342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:34.843966 342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:34.845477 342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:34.845561 342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:34.845686 342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:34.845738 342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:34.876713 342 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:40315
I20260929 11:27:34.876744 469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:40315 every 8 connection(s)
I20260929 11:27:34.878099 342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:27:34.881805 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 342
I20260929 11:27:34.881984 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:27:34.882529 24164 external_mini_cluster.cc:1428] Setting key c78f1b40dafc056516919d687e8fc2f4
Sep 29 11:27:34 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681254, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:27:34.901513 472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:34.881636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:38237 (local address 127.23.153.62:42209)
0929 11:27:34.882337 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:34.882343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:34.883491 (+ 1148us) server_negotiation.cc:407] Connection header received
0929 11:27:34.885778 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:34.885784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:34.885885 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:34.885994 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:34.887887 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:34.888682 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:34.890439 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:34.890870 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:34.890991 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:34.894993 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:34.894999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:34.895002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:34.895093 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:34.898036 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:34.898893 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:34.898906 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:34.898908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:34.899074 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:34.899572 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:34.899577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:34.899579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:34.899877 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:34.900043 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:34.900839 (+ 796us) server_negotiation.cc:299] Negotiation successful
0929 11:27:34.901116 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":406,"threads_started":1}
I20260929 11:27:34.902581 473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:34.882379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.3:38237)
0929 11:27:34.883334 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:34.883391 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:34.885173 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:34.886309 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:34.886320 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:34.886937 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:34.887663 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:34.887685 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:34.888877 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:34.888882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:34.890236 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:34.890253 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:34.891162 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:34.891171 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:34.891413 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:34.892200 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:34.892231 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:34.894690 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:34.898205 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:34.898214 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:34.898232 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:34.898733 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:34.899193 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:27:34.899197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:27:34.899200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:27:34.899424 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:27:34.900049 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:34.900058 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:34.900452 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 11:27:34.900716 (+ 264us) client_negotiation.cc:240] Negotiation successful
0929 11:27:34.901193 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":347,"threads_started":1}
I20260929 11:27:34.904165 470 heartbeater.cc:347] Connected to a master server at 127.23.153.62:42209
I20260929 11:27:34.904522 470 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:34.905321 470 heartbeater.cc:511] Master 127.23.153.62:42209 requested a full tablet report, sending...
I20260929 11:27:34.906893 32481 ts_manager.cc:194] Registered new tserver with Master: 48b959d3da724449800f64a1aeef9446 (127.23.153.3:40315)
I20260929 11:27:34.907790 32481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:38237
I20260929 11:27:34.911582 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.1.keytab.
Entry for principal oryx/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.1.keytab.
WARNING: no policy specified for oryx/127.23.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.2.keytab.
Entry for principal oryx/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.2.keytab.
WARNING: no policy specified for oryx/127.23.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.3.keytab.
Entry for principal oryx/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.3.keytab.
I20260929 11:27:35.058475 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 32542
Sep 29 11:27:35 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:27:35.076849 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:35.059847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.23.153.1:39593)
0929 11:27:35.060498 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:35.060505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:35.060536 (+ 31us) server_negotiation.cc:407] Connection header received
0929 11:27:35.060600 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:35.060607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:35.060979 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:35.061128 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:35.062585 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:35.063974 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:35.065553 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:35.066188 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:35.066322 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:35.070660 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:35.070676 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:35.070695 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:35.070747 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:35.073968 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:35.074837 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:35.074846 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:35.074848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:35.075059 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:35.075440 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:35.075443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:35.075445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:35.075809 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:35.076056 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:35.076314 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 11:27:35.076532 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":348,"threads_started":1}
I20260929 11:27:35.078123 32665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:35.162261 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 32542
I20260929 11:27:35.176924 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 32676
Sep 29 11:27:35 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.2@KRBTEST.COM
I20260929 11:27:35.194362 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:35.177827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60520 (local address 127.23.153.2:43843)
0929 11:27:35.178336 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:35.178343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:35.178367 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:27:35.178434 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:35.178441 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:35.178648 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:35.178777 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:35.180030 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:35.181074 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:35.182382 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:35.183004 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:35.183161 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:35.187907 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:35.187925 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:35.187944 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:35.187994 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:35.190932 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:35.192104 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:35.192112 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:35.192115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:35.192273 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:35.192685 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:35.192689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:35.192691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:35.193196 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:35.193463 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:35.193726 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 11:27:35.193955 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":264,"threads_started":1}
I20260929 11:27:35.195695 331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:35.279115 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 32676
I20260929 11:27:35.292325 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 342
Sep 29 11:27:35 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.3@KRBTEST.COM
I20260929 11:27:35.311239 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:35.293418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56326 (local address 127.23.153.3:40315)
0929 11:27:35.294105 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:35.294112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:35.294143 (+ 31us) server_negotiation.cc:407] Connection header received
0929 11:27:35.294208 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:35.294214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:35.294458 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:35.294600 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:35.296011 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:35.297114 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:35.298479 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:35.299151 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:35.299263 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:35.303847 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:35.303865 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:35.303885 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:35.303931 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:35.308278 (+ 4347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:35.309107 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:35.309115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:35.309118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:35.309275 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:35.309576 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:35.309578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:35.309580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:35.310031 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:35.310305 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:35.310588 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 11:27:35.310807 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":355,"threads_started":1}
I20260929 11:27:35.312713 465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:35.399581 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 342
I20260929 11:27:35.414759 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 32449
I20260929 11:27:35.416630 32511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:35.523895 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 32449
I20260929 11:27:35.543756 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:42209
--webserver_interface=127.23.153.62
--webserver_port=39877
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:44157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:42209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:35.897715 488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:35.898023 488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:35.898115 488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:35.905865 488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:27:35.906020 488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:35.906069 488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:35.906127 488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:27:35.906172 488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:27:35.917126 488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:44157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:42209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:42209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=39877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:35.918464 488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:35.919808 488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:35.932524 493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:35.932590 494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:35.932559 496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:35.933425 488 server_base.cc:1034] running on GCE node
I20260929 11:27:35.934315 488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:35.936133 488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:35.937382 488 hybrid_clock.cc:648] HybridClock initialized: now 1790681255937327 us; error 53 us; skew 500 ppm
I20260929 11:27:35.937767 488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:35 dist-test-slave-qk05 krb5kdc[32436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681255, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:35.941284 488 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:27:35.943228 488 webserver.cc:460] Webserver started at http://127.23.153.62:39877/ using document root <none> and password file <none>
I20260929 11:27:35.943984 488 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:35.944135 488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:35.951272 488 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 11:27:35.954828 503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:35.956503 488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:27:35.956696 488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "eb0449cb93c243b79fa0ffdcdc9fed42"
format_stamp: "Formatted at 2026-09-29 11:27:33 on dist-test-slave-qk05"
server_key: "f20c103ceda12ffd9ff27178a96ac0be"
server_key_iv: "a157234f7e2b45e836a6f6bbcf30575d"
server_key_version: "encryptionkey@0"
I20260929 11:27:35.957754 488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:35.975405 488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:35.980223 488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:35.980506 488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:35.999558 488 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:42209
I20260929 11:27:35.999567 554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:42209 every 8 connection(s)
I20260929 11:27:36.001195 488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:27:36.007660 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 488
I20260929 11:27:36.008529 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:39593
--local_ip_for_outbound_sockets=127.23.153.1
--tserver_master_addrs=127.23.153.62:42209
--webserver_port=39271
--webserver_interface=127.23.153.1
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:44157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.153.1
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:27:36.008603 555 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:27:36.012285 555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42: Bootstrap starting.
I20260929 11:27:36.032220 555 log.cc:824] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:36.042155 555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:27:36.042732 555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42: Bootstrap complete.
I20260929 11:27:36.051867 555 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0449cb93c243b79fa0ffdcdc9fed42" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42209 } }
I20260929 11:27:36.052809 555 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0449cb93c243b79fa0ffdcdc9fed42, State: Initialized, Role: FOLLOWER
I20260929 11:27:36.053488 555 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0449cb93c243b79fa0ffdcdc9fed42" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42209 } }
I20260929 11:27:36.053740 555 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:27:36.053859 555 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:27:36.054023 555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [term 1 FOLLOWER]: Advancing to term 2
I20260929 11:27:36.058212 555 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0449cb93c243b79fa0ffdcdc9fed42" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42209 } }
I20260929 11:27:36.058840 555 leader_election.cc:302] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb0449cb93c243b79fa0ffdcdc9fed42; no voters:
I20260929 11:27:36.059547 555 leader_election.cc:288] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 11:27:36.059693 559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 11:27:36.060930 559 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [term 2 LEADER]: Becoming Leader. State: Replica: eb0449cb93c243b79fa0ffdcdc9fed42, State: Running, Role: LEADER
I20260929 11:27:36.061651 559 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0449cb93c243b79fa0ffdcdc9fed42" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42209 } }
I20260929 11:27:36.062057 555 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:27:36.064253 561 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb0449cb93c243b79fa0ffdcdc9fed42. Latest consensus state: current_term: 2 leader_uuid: "eb0449cb93c243b79fa0ffdcdc9fed42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0449cb93c243b79fa0ffdcdc9fed42" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42209 } } }
I20260929 11:27:36.064669 561 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:36.065362 560 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eb0449cb93c243b79fa0ffdcdc9fed42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0449cb93c243b79fa0ffdcdc9fed42" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42209 } } }
I20260929 11:27:36.065680 560 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:36.075865 566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:27:36.082810 566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:27:36.084748 566 catalog_manager.cc:1259] Loaded cluster ID: a34ecc6b87314beb9e5236a2ab8241b2
I20260929 11:27:36.085003 566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:27:36.089355 566 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:27:36.091115 566 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eb0449cb93c243b79fa0ffdcdc9fed42: Loaded TSK: 0
I20260929 11:27:36.092864 566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:27:36.098268 576 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260929 11:27:36.387465 557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:36.387791 557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:36.387950 557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:36.396507 557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:36.396661 557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:36.396816 557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:27:36.408039 557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:44157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.1.keytab
--principal=oryx/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:39593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=39271
--tserver_master_addrs=127.23.153.62:42209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:36.409468 557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:36.410913 557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:36.424916 581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:36.424939 582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:36.425668 584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:36.426219 557 server_base.cc:1034] running on GCE node
I20260929 11:27:36.426940 557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:36.428287 557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:36.429541 557 hybrid_clock.cc:648] HybridClock initialized: now 1790681256429465 us; error 75 us; skew 500 ppm
I20260929 11:27:36.429934 557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:36 dist-test-slave-qk05 krb5kdc[32436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681256, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:36.433511 557 init.cc:377] Logged in from keytab as oryx/127.23.153.1@KRBTEST.COM (short username oryx)
I20260929 11:27:36.435290 557 webserver.cc:460] Webserver started at http://127.23.153.1:39271/ using document root <none> and password file <none>
I20260929 11:27:36.436189 557 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:36.436369 557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:36.442557 557 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 11:27:36.445928 591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:36.447616 557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 11:27:36.447830 557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "837f05ddce394df1b278e53ed317d6c1"
format_stamp: "Formatted at 2026-09-29 11:27:33 on dist-test-slave-qk05"
server_key: "e81e982ab60cb5054a7212205b395570"
server_key_iv: "1808ea48ab2129f65e4ad90b950ef546"
server_key_version: "encryptionkey@0"
I20260929 11:27:36.448889 557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:36.475188 557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:36.479892 557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:36.480156 557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:36.481225 557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:36.482693 557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:36.482795 557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:36.482903 557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:36.482975 557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:36.515739 557 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:39593
I20260929 11:27:36.515802 703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:39593 every 8 connection(s)
I20260929 11:27:36.517213 557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:27:36.523131 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 557
I20260929 11:27:36.523885 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:43843
--local_ip_for_outbound_sockets=127.23.153.2
--tserver_master_addrs=127.23.153.62:42209
--webserver_port=35543
--webserver_interface=127.23.153.2
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:44157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.153.2
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:27:36 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:36 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 11:27:36.884433 708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:36.884742 708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:36.884966 708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:36.892899 708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:36.893046 708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:36.893205 708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:27:36.903703 708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:44157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.2.keytab
--principal=oryx/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:43843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=35543
--tserver_master_addrs=127.23.153.62:42209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:36.905117 708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:36.906527 708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:36.922868 714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:36.922887 715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:36.923219 717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:36.923593 708 server_base.cc:1034] running on GCE node
I20260929 11:27:36.924427 708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:36.925478 708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:36.926704 708 hybrid_clock.cc:648] HybridClock initialized: now 1790681256926641 us; error 59 us; skew 500 ppm
I20260929 11:27:36.927050 708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:36 dist-test-slave-qk05 krb5kdc[32436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681256, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:36.930603 708 init.cc:377] Logged in from keytab as oryx/127.23.153.2@KRBTEST.COM (short username oryx)
I20260929 11:27:36.932365 708 webserver.cc:460] Webserver started at http://127.23.153.2:35543/ using document root <none> and password file <none>
I20260929 11:27:36.933104 708 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:36.933233 708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:36.939523 708 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 11:27:36.942320 724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:36.944240 708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:27:36.944437 708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "384bd752a12e4532867ee58b835e8f5f"
format_stamp: "Formatted at 2026-09-29 11:27:34 on dist-test-slave-qk05"
server_key: "1f060ca9db72aa3a01917d43252b2406"
server_key_iv: "202a544aed4d628b5ef47955bf2d4116"
server_key_version: "encryptionkey@0"
I20260929 11:27:36.945745 708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:36.959745 708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:36.964357 708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:36.964634 708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:36.965715 708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:36.967325 708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:36.967454 708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:36.967566 708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:36.967643 708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:37.018744 708 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:43843
I20260929 11:27:37.018797 836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:43843 every 8 connection(s)
I20260929 11:27:37.020579 708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:27:37.028946 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 708
I20260929 11:27:37.029683 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:40315
--local_ip_for_outbound_sockets=127.23.153.3
--tserver_master_addrs=127.23.153.62:42209
--webserver_port=46277
--webserver_interface=127.23.153.3
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:44157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.153.3
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:27:37 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:37 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 11:27:37.612495 841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:37.612949 841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:37.613174 841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:37.625579 841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:37.625742 841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:37.625976 841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:27:37.642730 841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:44157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/krb5kdc/oryx_127.23.153.3.keytab
--principal=oryx/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:40315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=46277
--tserver_master_addrs=127.23.153.62:42209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:37.644765 841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:37.646545 841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:37.666239 707 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.133s user 0.000s sys 0.013s
W20260929 11:27:37.667084 848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:37.668682 847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:37.668746 706 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:36.520302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:36867 (local address 127.23.153.62:42209)
0929 11:27:36.521135 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:36.521151 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:36.521367 (+ 216us) server_negotiation.cc:407] Connection header received
0929 11:27:36.524177 (+ 2810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:36.524194 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:36.524633 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:36.524983 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:36.527097 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:36.528552 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:36.530784 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:36.531567 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:36.531772 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:37.667628 (+1135856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.1:36867: BlockingRecv error: recv got EOF from 127.23.153.1:36867 (error 108)
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":285,"threads_started":1}
I20260929 11:27:37.671254 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:36.520363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.1:36867)
0929 11:27:36.521185 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:36.521236 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:36.523393 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:36.525242 (+ 1849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:36.525253 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:36.525981 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:36.526779 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:36.526807 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:36.528767 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:36.528773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:36.530597 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:36.530618 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:36.531845 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:36.531853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:36.532120 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:36.532774 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:36.532802 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:37.669972 (+1137170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":325,"threads_started":1}
W20260929 11:27:37.672914 704 heartbeater.cc:644] Failed to heartbeat to 127.23.153.62:42209 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.153.62:42209: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Sep 29 11:27:37 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:37 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 11:27:37.694540 706 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:37.674386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:49031 (local address 127.23.153.62:42209)
0929 11:27:37.674584 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:37.674591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:37.675499 (+ 908us) server_negotiation.cc:407] Connection header received
0929 11:27:37.675578 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:37.675584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:37.675672 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:37.675780 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:37.677304 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:37.678572 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:37.683750 (+ 5178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:37.684478 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:37.684598 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:37.694241 (+ 9643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.1:49031: BlockingRecv error: recv got EOF from 127.23.153.1:49031 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:27:37.696213 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:37.673891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.1:49031)
0929 11:27:37.675177 (+ 1286us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:37.675201 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:37.675372 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:37.676301 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:37.676307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:37.676843 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:37.677081 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:37.677094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:37.679834 (+ 2740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:37.679842 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:37.681544 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:37.681567 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:37.685010 (+ 3443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:37.685018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:37.685232 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:37.686498 (+ 1266us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:37.686524 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:37.695867 (+ 9343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1151,"thread_start_us":21637,"threads_started":1}
I20260929 11:27:37.705950 841 server_base.cc:1034] running on GCE node
W20260929 11:27:37.706382 850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:37.708153 841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
Sep 29 11:27:37 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:37.709815 841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Sep 29 11:27:37 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:37.711112 841 hybrid_clock.cc:648] HybridClock initialized: now 1790681257711018 us; error 74 us; skew 500 ppm
I20260929 11:27:37.711532 841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:27:37.713699 852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:37.697616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.1:46277)
0929 11:27:37.697912 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:37.697933 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:37.698147 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:37.698819 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:37.698825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:37.699533 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:37.699873 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:37.699888 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:37.701577 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:37.701584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:37.703313 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:37.703332 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:37.705342 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:37.705377 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:37.705659 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:37.706732 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:37.706757 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:37.713208 (+ 6451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166}
W20260929 11:27:37.713745 706 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:37.697250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:46277 (local address 127.23.153.62:42209)
0929 11:27:37.697455 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:37.697463 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:37.698027 (+ 564us) server_negotiation.cc:407] Connection header received
0929 11:27:37.698392 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:37.698399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:37.698499 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:37.698577 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:37.700089 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:37.701344 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:37.704308 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:37.705096 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:37.705278 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:37.713563 (+ 8285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.1:46277: BlockingRecv error: recv got EOF from 127.23.153.1:46277 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 11:27:37.714213 704 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 11:27:37 dist-test-slave-qk05 krb5kdc[32436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681257, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:37.716560 841 init.cc:377] Logged in from keytab as oryx/127.23.153.3@KRBTEST.COM (short username oryx)
I20260929 11:27:37.718417 841 webserver.cc:460] Webserver started at http://127.23.153.3:46277/ using document root <none> and password file <none>
I20260929 11:27:37.719262 841 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:37.719408 841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:37.725907 841 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:27:37.729416 858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:37.731910 841 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:27:37.732184 841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "48b959d3da724449800f64a1aeef9446"
format_stamp: "Formatted at 2026-09-29 11:27:34 on dist-test-slave-qk05"
server_key: "eda5316af0d62f4f3cbbb74254a5e8de"
server_key_iv: "0ec36d8902c09490859853de3e0e7993"
server_key_version: "encryptionkey@0"
I20260929 11:27:37.733937 841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:37.777827 841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:37.785219 841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:37.785671 841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:37.787370 841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:37.789832 841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:37.789999 841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:37.790143 841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:37.790235 841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:37.829198 841 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:40315
I20260929 11:27:37.829269 970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:40315 every 8 connection(s)
I20260929 11:27:37.830957 841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:27:37.832475 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 841
Sep 29 11:27:37 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:37 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:37.865810 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:37.834231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.23.153.62:42209)
0929 11:27:37.834581 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:37.834588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:37.845106 (+ 10518us) server_negotiation.cc:407] Connection header received
0929 11:27:37.845691 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:37.845698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:37.845811 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:37.845899 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:37.848429 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:37.849457 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:37.851837 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:37.852424 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:37.852541 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:37.856141 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:37.856155 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:37.856186 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:37.856255 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:37.860931 (+ 4676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:37.862006 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:37.862016 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:37.862028 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:37.862389 (+ 361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:27:37.862967 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:27:37.862973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:27:37.862977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:27:37.863889 (+ 912us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:37.864281 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:37.864858 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 11:27:37.865373 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
W20260929 11:27:38.092869 840 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.054s user 0.008s sys 0.004s
W20260929 11:27:38.094540 839 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:37.024458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:51865 (local address 127.23.153.62:42209)
0929 11:27:37.025048 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:37.025055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:37.026137 (+ 1082us) server_negotiation.cc:407] Connection header received
0929 11:27:37.028713 (+ 2576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:37.028720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:37.028836 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:37.028927 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:37.031715 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:37.033251 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:37.036014 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:37.036842 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:37.037223 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:38.094121 (+1056898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.2:51865: BlockingRecv error: recv got EOF from 127.23.153.2:51865 (error 108)
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":279,"threads_started":1}
I20260929 11:27:38.094825 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:37.025012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.2:51865)
0929 11:27:37.025937 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:37.025990 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:37.028301 (+ 2311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:37.029604 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:37.029618 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:37.030434 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:37.031375 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:37.031403 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:37.033563 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:37.033570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:37.035755 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:37.035780 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:37.037509 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:37.037536 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:37.038030 (+ 494us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:37.038916 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:37.038945 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:38.093613 (+1054668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":338,"threads_started":1}
W20260929 11:27:38.096045 837 heartbeater.cc:644] Failed to heartbeat to 127.23.153.62:42209 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.153.62:42209: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Sep 29 11:27:38 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:38 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:38.112609 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:38.096853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.2:50571)
0929 11:27:38.097047 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:38.097069 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:38.097199 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:38.097642 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:38.097646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:38.098013 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:38.098245 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:38.098258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.100174 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.100179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:38.101541 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:38.101561 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.102759 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.102765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:38.102917 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:38.107243 (+ 4326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:38.107266 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:38.112378 (+ 5112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 11:27:38.113497 839 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:38.096975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:50571 (local address 127.23.153.62:42209)
0929 11:27:38.097159 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:38.097165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:38.097183 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:27:38.097341 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:38.097346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:38.097500 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:38.097586 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:38.098470 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.099925 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.101742 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.102488 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.102671 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:38.113172 (+ 10501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.2:50571: BlockingRecv error: recv got EOF from 127.23.153.2:50571 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 11:27:38 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:38 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:38.123293 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:38.113453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.2:47857)
0929 11:27:38.113699 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:38.113716 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:38.113914 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:38.114316 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:38.114321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:38.114651 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:38.114824 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:38.114833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.115979 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.115983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:38.116967 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:38.116981 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.117745 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.117751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:38.117960 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:38.118738 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:38.118761 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:38.123056 (+ 4295us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149}
W20260929 11:27:38.123629 837 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 11:27:38.123865 706 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:38.113699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:47857 (local address 127.23.153.62:42209)
0929 11:27:38.113952 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:38.113958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:38.113999 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:27:38.114059 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:38.114064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:38.114166 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:38.114261 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:38.114977 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.115826 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.117129 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.117557 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.117695 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:38.123396 (+ 5701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.2:47857: BlockingRecv error: recv got EOF from 127.23.153.2:47857 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 29 11:27:38 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:38 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:38.730404 977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:38.714953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.1:41783)
0929 11:27:38.718995 (+ 4042us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:38.719056 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:38.719228 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:38.719614 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:38.719619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:38.720251 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:38.720528 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:38.720542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.721781 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.721787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:38.723596 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:38.723622 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.724455 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.724461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:38.724661 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:38.725405 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:38.725433 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:38.730095 (+ 4662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3884,"thread_start_us":3746,"threads_started":1}
W20260929 11:27:38.731272 978 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:38.715188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:41783 (local address 127.23.153.62:42209)
0929 11:27:38.719297 (+ 4109us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:38.719301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:38.719320 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:27:38.719369 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:38.719373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:38.719434 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:38.719537 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:38.720799 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.721605 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.723836 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.724264 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.724382 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:38.730956 (+ 6574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.1:41783: BlockingRecv error: recv got EOF from 127.23.153.1:41783 (error 108)
Metrics: {"server-negotiator.queue_time_us":4000,"thread_start_us":3872,"threads_started":1}
W20260929 11:27:38.883438 975 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.028s user 0.004s sys 0.010s
W20260929 11:27:38.884722 974 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:37.834324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:45975 (local address 127.23.153.62:42209)
0929 11:27:37.847426 (+ 13102us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:37.847430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:37.847446 (+ 16us) server_negotiation.cc:407] Connection header received
0929 11:27:37.847492 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:37.847496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:37.847552 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:37.847657 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:37.849775 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:37.850566 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:37.852555 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:37.853454 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:37.853594 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:38.884477 (+1030883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.3:45975: BlockingRecv error: recv got EOF from 127.23.153.3:45975 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":13007,"thread_start_us":12870,"threads_started":1}
I20260929 11:27:38.885129 975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:37.835208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.3:45975)
0929 11:27:37.843735 (+ 8527us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:37.843792 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:37.846542 (+ 2750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:37.848013 (+ 1471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:37.848024 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:37.848707 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:37.849587 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:37.849618 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:37.850728 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:37.850733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:37.852377 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:37.852397 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:37.853752 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:37.853761 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:37.854092 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:37.854855 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:37.854890 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:38.884200 (+1029310us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":8159,"thread_start_us":7894,"threads_started":1}
W20260929 11:27:38.886171 971 heartbeater.cc:644] Failed to heartbeat to 127.23.153.62:42209 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.153.62:42209: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Sep 29 11:27:38 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:38 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:38.896368 975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:38.887046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.3:40449)
0929 11:27:38.887267 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:38.887288 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:38.887415 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:38.887759 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:38.887763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:38.888192 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:38.888376 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:38.888388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.889553 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.889557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:38.890544 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:38.890556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.891618 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.891628 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:38.891772 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:38.892387 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:38.892408 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:38.896063 (+ 3655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 11:27:38.896556 974 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:38.887118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:40449 (local address 127.23.153.62:42209)
0929 11:27:38.887371 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:38.887375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:38.887391 (+ 16us) server_negotiation.cc:407] Connection header received
0929 11:27:38.887528 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:38.887545 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:38.887606 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:38.887700 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:38.888519 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.889409 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.890705 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.891180 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.891296 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:38.896348 (+ 5052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.3:40449: BlockingRecv error: recv got EOF from 127.23.153.3:40449 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
Sep 29 11:27:38 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:38 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:38.906394 975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:38.897276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.3:34889)
0929 11:27:38.897433 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:38.897453 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:38.897574 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:38.897921 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:38.897926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:38.898271 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:38.898443 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:38.898453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.899643 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.899648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:38.901041 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:38.901057 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.901996 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.902001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:38.902139 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:38.902729 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:38.902747 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:38.906156 (+ 3409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 11:27:38.906697 974 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:38.897386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:34889 (local address 127.23.153.62:42209)
0929 11:27:38.897542 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:38.897548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:38.897566 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:27:38.897692 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:38.897697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:38.897785 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:38.897867 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:38.898588 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.899511 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.901363 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:38.901825 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:38.901930 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:38.906403 (+ 4473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.3:34889: BlockingRecv error: recv got EOF from 127.23.153.3:34889 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 11:27:38.906711 971 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 11:27:39 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:39 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:39.133957 979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:39.124417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.2:56637)
0929 11:27:39.125049 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:39.125065 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:39.125198 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:39.125588 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:39.125592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:39.126034 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:39.126186 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:39.126195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:39.127338 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:39.127343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:39.128355 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:39.128370 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:39.129214 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:39.129222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:39.129401 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:39.130049 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:39.130070 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:39.133577 (+ 3507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":392,"threads_started":1}
W20260929 11:27:39.134104 974 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:39.124524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:56637 (local address 127.23.153.62:42209)
0929 11:27:39.124723 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:39.124728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:39.125166 (+ 438us) server_negotiation.cc:407] Connection header received
0929 11:27:39.125300 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:39.125305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:39.125392 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:39.125488 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:39.126396 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:39.127187 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:39.128499 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:39.129005 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:39.129151 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:39.133865 (+ 4714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.2:56637: BlockingRecv error: recv got EOF from 127.23.153.2:56637 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 11:27:39 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:39 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:39.743362 980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:39.731832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.1:50215)
0929 11:27:39.732338 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:39.732356 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:39.732487 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:39.733030 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:39.733034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:39.733535 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:39.733714 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:39.733725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:39.735158 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:39.735165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:39.736301 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:39.736318 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:39.737205 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:39.737216 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:39.737364 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:39.737991 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:39.738022 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:39.742607 (+ 4585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":264,"threads_started":1}
W20260929 11:27:39.743640 981 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:39.731976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:50215 (local address 127.23.153.62:42209)
0929 11:27:39.732666 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:39.732672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:39.732693 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:27:39.732761 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:39.732766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:39.732839 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:39.732963 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:39.733874 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:39.734870 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:39.736498 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:39.736936 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:39.737050 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:39.743259 (+ 6209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.1:50215: BlockingRecv error: recv got EOF from 127.23.153.1:50215 (error 108)
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":447,"threads_started":1}
Sep 29 11:27:39 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:39 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:39.917449 982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:39.907395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.3:50367)
0929 11:27:39.907923 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:39.907941 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:39.908068 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:39.908391 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:39.908394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:39.908794 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:39.908946 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:39.908955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:39.909985 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:39.909990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:39.910979 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:39.910992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:39.911926 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:39.911932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:39.912098 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:39.912720 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:39.912739 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:39.917058 (+ 4319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":292,"threads_started":1}
W20260929 11:27:39.917760 981 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:39.907604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:50367 (local address 127.23.153.62:42209)
0929 11:27:39.907835 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:39.907841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:39.908018 (+ 177us) server_negotiation.cc:407] Connection header received
0929 11:27:39.908179 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:39.908184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:39.908251 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:39.908333 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:39.909063 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:39.909841 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:39.911203 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:39.911713 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:39.911825 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:39.917485 (+ 5660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.3:50367: BlockingRecv error: recv got EOF from 127.23.153.3:50367 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 29 11:27:40 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:40 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:40.147632 983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:40.135234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.2:49399)
0929 11:27:40.135794 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:40.135810 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:40.135940 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:40.136485 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:40.136494 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:40.137066 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:40.137319 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:40.137331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:40.138785 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:40.138792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:40.139899 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:40.139922 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:40.141077 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:40.141083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:40.141283 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:40.142395 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:40.142439 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:40.147353 (+ 4914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":371,"threads_started":1}
W20260929 11:27:40.147866 981 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:40.135334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:49399 (local address 127.23.153.62:42209)
0929 11:27:40.135574 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:40.135580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:40.135899 (+ 319us) server_negotiation.cc:407] Connection header received
0929 11:27:40.136041 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:40.136045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:40.136130 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:40.136223 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:40.137594 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:40.138467 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:40.140221 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:40.140830 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:40.140998 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:40.147623 (+ 6625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.2:49399: BlockingRecv error: recv got EOF from 127.23.153.2:49399 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 29 11:27:40 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:40 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:40.755143 984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:40.744824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.1:33533)
0929 11:27:40.745348 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:40.745370 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:40.745518 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:40.746064 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:40.746068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:40.746551 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:40.746774 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:40.746792 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:40.748140 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:40.748145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:40.749105 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:40.749120 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:40.749890 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:40.749895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:40.750041 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:40.750900 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:40.750921 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:40.754584 (+ 3663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":265,"threads_started":1}
W20260929 11:27:40.755504 985 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:40.745057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:33533 (local address 127.23.153.62:42209)
0929 11:27:40.745732 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:40.745738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:40.745755 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:27:40.745804 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:40.745809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:40.745880 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:40.745977 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:40.746926 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:40.747978 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:40.749271 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:40.749727 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:40.749833 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:40.755209 (+ 5376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.1:33533: BlockingRecv error: recv got EOF from 127.23.153.1:33533 (error 108)
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":400,"threads_started":1}
Sep 29 11:27:40 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:40 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:40.928122 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:40.918601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.3:39239)
0929 11:27:40.919139 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:40.919156 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:40.919294 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:40.919627 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:40.919632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:40.920067 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:40.920250 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:40.920258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:40.921287 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:40.921293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:40.922219 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:40.922231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:40.922929 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:40.922935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:40.923145 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:40.923808 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:40.923831 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:40.927776 (+ 3945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":279,"threads_started":1}
W20260929 11:27:40.928224 985 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:40.918770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:39239 (local address 127.23.153.62:42209)
0929 11:27:40.918993 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:40.918999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:40.919242 (+ 243us) server_negotiation.cc:407] Connection header received
0929 11:27:40.919428 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:40.919432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:40.919493 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:40.919576 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:40.920399 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:40.921148 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:40.922367 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:40.922769 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:40.922884 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:40.928019 (+ 5135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.3:39239: BlockingRecv error: recv got EOF from 127.23.153.3:39239 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 11:27:41 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:41 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:41.158304 987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:41.148747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.2:39383)
0929 11:27:41.149339 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:41.149355 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:41.149499 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:41.149868 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:41.149873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:41.150273 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:41.150484 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:41.150495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:41.151623 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:41.151629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:41.152627 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:41.152644 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:41.153392 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:41.153398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:41.153581 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:41.154281 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:41.154307 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:41.158058 (+ 3751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":299,"threads_started":1}
W20260929 11:27:41.158519 985 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:41.148915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:39383 (local address 127.23.153.62:42209)
0929 11:27:41.149156 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:41.149162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:41.149456 (+ 294us) server_negotiation.cc:407] Connection header received
0929 11:27:41.149634 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:41.149638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:41.149715 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:41.149801 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:41.150635 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:41.151472 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:41.152798 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:41.153235 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:41.153358 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:41.158262 (+ 4904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.2:39383: BlockingRecv error: recv got EOF from 127.23.153.2:39383 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 29 11:27:41 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:41 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:41.766723 988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:41.756239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.1:49383)
0929 11:27:41.757127 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:41.757145 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:41.757274 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:41.757728 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:41.757733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:41.758238 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:41.758409 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:41.758420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:41.759737 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:41.759742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:41.760869 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:41.760888 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:41.761805 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:41.761812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:41.761984 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:41.762591 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:41.762635 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:41.766445 (+ 3810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":659,"threads_started":1}
W20260929 11:27:41.767005 989 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:41.756314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:49383 (local address 127.23.153.62:42209)
0929 11:27:41.756865 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:41.756872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:41.757344 (+ 472us) server_negotiation.cc:407] Connection header received
0929 11:27:41.757420 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:41.757430 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:41.757515 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:41.757628 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:41.758698 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:41.759549 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:41.761062 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:41.761631 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:41.761735 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:41.766744 (+ 5009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.1:49383: BlockingRecv error: recv got EOF from 127.23.153.1:49383 (error 108)
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":301,"threads_started":1}
Sep 29 11:27:41 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:41 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:41.938774 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:41.929252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.3:48397)
0929 11:27:41.929782 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:41.929801 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:41.929927 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:41.930310 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:41.930314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:41.930674 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:41.930864 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:41.930873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:41.931930 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:41.931934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:41.932859 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:41.932872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:41.933572 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:41.933577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:41.933719 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:41.934335 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:41.934358 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:41.938418 (+ 4060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":296,"threads_started":1}
W20260929 11:27:41.938875 989 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:41.929383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:48397 (local address 127.23.153.62:42209)
0929 11:27:41.929635 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:41.929640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:41.929880 (+ 240us) server_negotiation.cc:407] Connection header received
0929 11:27:41.930022 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:41.930026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:41.930098 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:41.930241 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:41.931042 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:41.931788 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:41.933011 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:41.933427 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:41.933520 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:41.938659 (+ 5139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.3:48397: BlockingRecv error: recv got EOF from 127.23.153.3:48397 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
Sep 29 11:27:42 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:42 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:42.168606 991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:42.159417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.2:36489)
0929 11:27:42.159920 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:42.159936 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:42.160080 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:42.160526 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:42.160531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:42.160995 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:42.161198 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:42.161208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:42.162394 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:42.162398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:42.163492 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:42.163504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:42.164241 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:42.164245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:42.164384 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:42.164960 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:42.164978 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:42.168351 (+ 3373us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":271,"threads_started":1}
W20260929 11:27:42.168831 989 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:42.159552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:36489 (local address 127.23.153.62:42209)
0929 11:27:42.159777 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:42.159784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:42.160028 (+ 244us) server_negotiation.cc:407] Connection header received
0929 11:27:42.160202 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:42.160207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:42.160310 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:42.160439 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:42.161328 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:42.162270 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:42.163627 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:42.164121 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:42.164220 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:42.168585 (+ 4365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.2:36489: BlockingRecv error: recv got EOF from 127.23.153.2:36489 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 29 11:27:42 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:42 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:42.776310 992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:42.767553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.1:58313)
0929 11:27:42.768139 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:42.768156 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:42.768268 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:42.768560 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:42.768564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:42.768926 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:42.769075 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:42.769084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:42.770089 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:42.770092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:42.771100 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:42.771115 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:42.771882 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:42.771887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:42.772032 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:42.772599 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:42.772617 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:42.776077 (+ 3460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":358,"threads_started":1}
W20260929 11:27:42.776649 993 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:42.767668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:58313 (local address 127.23.153.62:42209)
0929 11:27:42.768165 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:42.768169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:42.768210 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:27:42.768367 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:42.768370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:42.768429 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:42.768529 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:42.769191 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:42.769955 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:42.771252 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:42.771667 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:42.771769 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:42.776351 (+ 4582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.1:58313: BlockingRecv error: recv got EOF from 127.23.153.1:58313 (error 108)
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":274,"threads_started":1}
Sep 29 11:27:42 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:42 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:42.949712 994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:42.940143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.3:37861)
0929 11:27:42.940720 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:42.940740 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:42.940875 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:42.941283 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:42.941288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:42.941903 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:42.942119 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:42.942130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:42.943188 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:42.943194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:42.944311 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:42.944328 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:42.945088 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:42.945093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:42.945239 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:42.945806 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:42.945823 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:42.949435 (+ 3612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":265,"threads_started":1}
W20260929 11:27:42.949939 993 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:42.940293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:37861 (local address 127.23.153.62:42209)
0929 11:27:42.940501 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:42.940507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:42.940835 (+ 328us) server_negotiation.cc:407] Connection header received
0929 11:27:42.940995 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:42.941000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:42.941098 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:42.941220 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:42.942250 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:42.942978 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:42.944473 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:42.944943 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:42.945047 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:42.949695 (+ 4648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.3:37861: BlockingRecv error: recv got EOF from 127.23.153.3:37861 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 11:27:43 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:43 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 11:27:43.182946 993 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:43.169800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:57261 (local address 127.23.153.62:42209)
0929 11:27:43.170033 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:43.170039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:43.170188 (+ 149us) server_negotiation.cc:407] Connection header received
0929 11:27:43.170372 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:43.170377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:43.170453 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:43.170533 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:43.171305 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:43.172054 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:43.173292 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:43.173697 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:43.173814 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:43.182698 (+ 8884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.2:57261: BlockingRecv error: recv got EOF from 127.23.153.2:57261 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 11:27:43.183060 995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:43.169563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.2:57261)
0929 11:27:43.170096 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:43.170111 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:43.170245 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:43.170585 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:43.170590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:43.170986 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:43.171163 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:43.171172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:43.172188 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:43.172194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:43.173142 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:43.173154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:43.173844 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:43.173849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:43.174031 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:43.178456 (+ 4425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:43.178487 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:43.182550 (+ 4063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":282,"threads_started":1}
Sep 29 11:27:43 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:43 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:43.786553 996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:43.777379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.1:49689)
0929 11:27:43.777902 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:43.777923 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:43.778044 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:43.778416 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:43.778419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:43.778849 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:43.779072 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:43.779084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:43.780070 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:43.780074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:43.780976 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:43.780987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:43.781738 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:43.781744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:43.781929 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:43.782696 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:43.782714 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:43.786293 (+ 3579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":238,"threads_started":1}
W20260929 11:27:43.786778 997 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:43.777471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:49689 (local address 127.23.153.62:42209)
0929 11:27:43.777970 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:43.777974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:43.777994 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:27:43.778191 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:43.778196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:43.778285 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:43.778387 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:43.779189 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:43.779922 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:43.781131 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:43.781583 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:43.781695 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:43.786540 (+ 4845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.1:49689: BlockingRecv error: recv got EOF from 127.23.153.1:49689 (error 108)
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":247,"threads_started":1}
Sep 29 11:27:43 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:43 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:43.960831 998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:43.950821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.3:43979)
0929 11:27:43.951528 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:43.951545 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:43.951675 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:43.952047 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:43.952051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:43.952665 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:43.952818 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:43.952828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:43.954018 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:43.954023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:43.955240 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:43.955260 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:43.956183 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:43.956264 (+ 81us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:43.956463 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:43.957095 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:43.957114 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:43.960580 (+ 3466us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":412,"threads_started":1}
W20260929 11:27:43.961050 997 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:43.951002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:43979 (local address 127.23.153.62:42209)
0929 11:27:43.951390 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:43.951413 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:43.951630 (+ 217us) server_negotiation.cc:407] Connection header received
0929 11:27:43.951811 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:43.951815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:43.951883 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:43.951976 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:43.952979 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:43.953841 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:43.955474 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:43.955981 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:43.956135 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:43.960823 (+ 4688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.3:43979: BlockingRecv error: recv got EOF from 127.23.153.3:43979 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 11:27:44 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:44 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:44.192895 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:44.184165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.2:35153)
0929 11:27:44.184649 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:44.184666 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:44.184801 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:44.185138 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:44.185142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:44.185549 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:44.185720 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:44.185729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:44.186720 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:44.186726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:44.187698 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:44.187711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:44.188419 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:44.188425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:44.188592 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:44.189170 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:44.189188 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:44.192657 (+ 3469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":258,"threads_started":1}
W20260929 11:27:44.193148 997 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:44.184335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:35153 (local address 127.23.153.62:42209)
0929 11:27:44.184526 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:44.184532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:44.184760 (+ 228us) server_negotiation.cc:407] Connection header received
0929 11:27:44.184927 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:44.184933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:44.185010 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:44.185091 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:44.185865 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:44.186582 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:44.187852 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:44.188265 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:44.188377 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:44.192896 (+ 4519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.2:35153: BlockingRecv error: recv got EOF from 127.23.153.2:35153 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 11:27:44 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:44 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:44.799420 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:44.787693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.1:53625)
0929 11:27:44.788285 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:44.788302 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:44.788418 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:44.788940 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:44.788944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:44.789339 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:44.789515 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:44.789525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:44.790740 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:44.790747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:44.791971 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:44.791994 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:44.793065 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:44.793080 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:44.793435 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:44.794150 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:44.794173 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:44.799113 (+ 4940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":269,"threads_started":1}
W20260929 11:27:44.799775 1001 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:44.787801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:53625 (local address 127.23.153.62:42209)
0929 11:27:44.788613 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:44.788618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:44.788635 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:27:44.788689 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:44.788693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:44.788757 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:44.788855 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:44.789711 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:44.790470 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:44.792209 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:44.792773 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:44.792946 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:44.799465 (+ 6519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.1:53625: BlockingRecv error: recv got EOF from 127.23.153.1:53625 (error 108)
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":616,"threads_started":1}
Sep 29 11:27:44 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:44 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:44.970899 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:44.961910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.3:57413)
0929 11:27:44.962434 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:44.962451 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:44.962597 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:44.962940 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:44.962943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:44.963342 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:44.963510 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:44.963519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:44.964629 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:44.964633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:44.965579 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:44.965591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:44.966369 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:44.966373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:44.966508 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:44.967137 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:44.967156 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:44.970569 (+ 3413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":288,"threads_started":1}
W20260929 11:27:44.970949 1001 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:44.962013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:57413 (local address 127.23.153.62:42209)
0929 11:27:44.962171 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:44.962176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:44.962538 (+ 362us) server_negotiation.cc:407] Connection header received
0929 11:27:44.962734 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:44.962740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:44.962817 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:44.962911 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:44.963687 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:44.964482 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:44.965728 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:44.966212 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:44.966311 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:44.970776 (+ 4465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.3:57413: BlockingRecv error: recv got EOF from 127.23.153.3:57413 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 11:27:45 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:45 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 11:27:45.204174 1001 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:45.194466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:38405 (local address 127.23.153.62:42209)
0929 11:27:45.194765 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:45.194771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:45.195194 (+ 423us) server_negotiation.cc:407] Connection header received
0929 11:27:45.195386 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:45.195390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:45.195472 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:45.195558 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:45.196436 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:45.197191 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:45.198485 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:45.198950 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:45.199132 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:45.203937 (+ 4805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.2:38405: BlockingRecv error: recv got EOF from 127.23.153.2:38405 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
I20260929 11:27:45.204160 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:45.194281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.2:38405)
0929 11:27:45.195066 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:45.195088 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:45.195260 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:45.195620 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:45.195624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:45.196097 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:45.196274 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:45.196285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:45.197345 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:45.197351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:45.198317 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:45.198329 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:45.199172 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:45.199179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:45.199366 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:45.200023 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:45.200043 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:45.203683 (+ 3640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":402,"threads_started":1}
Sep 29 11:27:45 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:45 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:45.810659 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:45.800609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.1:34945)
0929 11:27:45.801258 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:45.801278 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:45.801404 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:45.801853 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:45.801857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:45.802414 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:45.802612 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:45.802623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:45.803792 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:45.803796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:45.804771 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:45.804785 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:45.805621 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:45.805627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:45.805753 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:45.806458 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:45.806482 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:45.810290 (+ 3808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":284,"threads_started":1}
W20260929 11:27:45.810774 1005 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:45.800766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:34945 (local address 127.23.153.62:42209)
0929 11:27:45.801469 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:45.801473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:45.801491 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:27:45.801556 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:45.801560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:45.801621 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:45.801821 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:45.802778 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:45.803657 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:45.804922 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:45.805422 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:45.805520 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:45.810537 (+ 5017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.1:34945: BlockingRecv error: recv got EOF from 127.23.153.1:34945 (error 108)
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":487,"threads_started":1}
Sep 29 11:27:45 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:45 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:45.981642 1006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:45.971854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.3:57597)
0929 11:27:45.972567 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:45.972584 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:45.972704 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:45.972984 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:45.972987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:45.973514 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:45.973703 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:45.973714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:45.974781 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:45.974785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:45.975798 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:45.975813 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:45.976618 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:45.976624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:45.976760 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:45.977637 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:45.977658 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:45.981378 (+ 3720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":399,"threads_started":1}
W20260929 11:27:45.981843 1005 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:45.972019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:57597 (local address 127.23.153.62:42209)
0929 11:27:45.972219 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:45.972224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:45.972675 (+ 451us) server_negotiation.cc:407] Connection header received
0929 11:27:45.972800 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:45.972804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:45.972870 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:45.972955 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:45.973841 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:45.974623 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:45.975961 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:45.976415 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:45.976523 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:45.981632 (+ 5109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.3:57597: BlockingRecv error: recv got EOF from 127.23.153.3:57597 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 11:27:46 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:46 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:46.215642 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:46.205429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.2:53479)
0929 11:27:46.206002 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:46.206017 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:46.206146 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:46.206679 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:46.206682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:46.207171 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:46.207356 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:46.207365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:46.208650 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:46.208655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:46.209634 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:46.209648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:46.210518 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:46.210524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:46.210651 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:46.211345 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:46.211366 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:46.215289 (+ 3923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":308,"threads_started":1}
W20260929 11:27:46.215761 1005 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:46.205603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:53479 (local address 127.23.153.62:42209)
0929 11:27:46.205772 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:46.205776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:46.206176 (+ 400us) server_negotiation.cc:407] Connection header received
0929 11:27:46.206304 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:46.206309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:46.206503 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:46.206609 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:46.207533 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:46.208474 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:46.209839 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:46.210309 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:46.210421 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:46.215526 (+ 5105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.2:53479: BlockingRecv error: recv got EOF from 127.23.153.2:53479 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 11:27:46 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:46 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:46.821278 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:46.811787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.1:58863)
0929 11:27:46.812296 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:46.812313 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:46.812421 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:46.812736 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:46.812739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:46.813127 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:46.813285 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:46.813293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:46.814279 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:46.814283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:46.815293 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:46.815307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:46.816146 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:46.816152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:46.816298 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:46.816885 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:46.816904 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:46.820969 (+ 4065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":273,"threads_started":1}
W20260929 11:27:46.821460 1009 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:46.811916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:58863 (local address 127.23.153.62:42209)
0929 11:27:46.812413 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:46.812418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:46.812437 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:27:46.812514 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:46.812524 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:46.812595 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:46.812695 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:46.813414 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:46.814164 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:46.815486 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:46.815959 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:46.816075 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:46.821208 (+ 5133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.1:58863: BlockingRecv error: recv got EOF from 127.23.153.1:58863 (error 108)
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":261,"threads_started":1}
Sep 29 11:27:46 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:46 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:46.992727 1010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:46.983091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.3:49173)
0929 11:27:46.983642 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:46.983658 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:46.983804 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:46.984129 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:46.984133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:46.984549 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:46.984712 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:46.984723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:46.985907 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:46.985912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:46.986859 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:46.986872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:46.987676 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:46.987684 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:46.987858 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:46.988504 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:46.988523 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:46.992457 (+ 3934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":286,"threads_started":1}
W20260929 11:27:46.992930 1009 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:46.983235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:49173 (local address 127.23.153.62:42209)
0929 11:27:46.983483 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:46.983489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:46.983747 (+ 258us) server_negotiation.cc:407] Connection header received
0929 11:27:46.983923 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:46.983927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:46.983991 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:46.984075 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:46.984877 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:46.985773 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:46.987002 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:46.987467 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:46.987595 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:46.992718 (+ 5123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.3:49173: BlockingRecv error: recv got EOF from 127.23.153.3:49173 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 29 11:27:47 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:47 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:47.226824 1011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:47.216964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.2:39819)
0929 11:27:47.217512 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:47.217528 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:47.217647 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:47.218035 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:47.218039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:47.218439 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:47.218585 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:47.218593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:47.219625 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:47.219629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:47.220700 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:47.220715 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:47.221539 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:47.221544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:47.221681 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:47.222416 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:47.222436 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:47.226571 (+ 4135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":304,"threads_started":1}
W20260929 11:27:47.227106 1009 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:47.217133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:39819 (local address 127.23.153.62:42209)
0929 11:27:47.217304 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:47.217308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:47.217601 (+ 293us) server_negotiation.cc:407] Connection header received
0929 11:27:47.217747 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:47.217751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:47.217847 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:47.217943 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:47.218715 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:47.219502 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:47.220928 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:47.221384 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:47.221492 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:47.226818 (+ 5326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.2:39819: BlockingRecv error: recv got EOF from 127.23.153.2:39819 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 11:27:47 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:47 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:47.832752 1013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:47.822424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.1:35733)
0929 11:27:47.823285 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:47.823303 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:47.823442 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:47.823787 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:47.823790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:47.824415 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:47.824759 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:47.824777 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:47.825869 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:47.825873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:47.826979 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:47.826994 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:47.827809 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:47.827816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:47.827994 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:47.828614 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:47.828632 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:47.832418 (+ 3786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":471,"threads_started":1}
W20260929 11:27:47.832901 1012 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:47.822481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:35733 (local address 127.23.153.62:42209)
0929 11:27:47.823213 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:47.823220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:47.823420 (+ 200us) server_negotiation.cc:407] Connection header received
0929 11:27:47.823544 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:47.823548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:47.823612 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:47.823711 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:47.824955 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:47.825725 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:47.827201 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:47.827649 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:47.827766 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:47.832666 (+ 4900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.1:35733: BlockingRecv error: recv got EOF from 127.23.153.1:35733 (error 108)
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":450,"threads_started":1}
Sep 29 11:27:47 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:47 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:48.004849 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:47.993976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.3:55977)
0929 11:27:47.994790 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:47.994807 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:47.994926 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:47.995291 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:47.995296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:47.995701 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:47.995851 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:47.995860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:47.996892 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:47.996896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:47.997847 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:47.997860 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:47.998775 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:47.998782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:47.998965 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:47.999613 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:47.999632 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:48.004341 (+ 4709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":550,"threads_started":1}
W20260929 11:27:48.005048 1012 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:47.994293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:55977 (local address 127.23.153.62:42209)
0929 11:27:47.994562 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:47.994569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:47.994894 (+ 325us) server_negotiation.cc:407] Connection header received
0929 11:27:47.995091 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:47.995095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:47.995157 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:47.995237 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:47.996001 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:47.996749 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:47.998016 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:47.998581 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:47.998729 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:48.004707 (+ 5978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.3:55977: BlockingRecv error: recv got EOF from 127.23.153.3:55977 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 29 11:27:48 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:48 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:48.243191 1015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:48.227974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.2:54689)
0929 11:27:48.228615 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:48.228630 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:48.228785 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:48.229121 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:48.229127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:48.229516 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:48.229673 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:48.229683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:48.230702 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:48.230708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:48.231836 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:48.231850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:48.232562 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:48.232568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:48.232725 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:48.237952 (+ 5227us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:48.237978 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:48.242800 (+ 4822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":401,"threads_started":1}
W20260929 11:27:48.243374 1012 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:48.228152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:54689 (local address 127.23.153.62:42209)
0929 11:27:48.228490 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:48.228496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:48.228721 (+ 225us) server_negotiation.cc:407] Connection header received
0929 11:27:48.228909 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:48.228913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:48.228989 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:48.229073 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:48.229834 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:48.230560 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:48.231989 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:48.232405 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:48.232516 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:48.243121 (+ 10605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.2:54689: BlockingRecv error: recv got EOF from 127.23.153.2:54689 (error 108)
Metrics: {"server-negotiator.queue_time_us":181}
Sep 29 11:27:48 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:48 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:48.843794 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:48.833792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.1:40163)
0929 11:27:48.834321 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:48.834345 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:48.834479 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:48.834852 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:48.834856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:48.835295 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:48.835482 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:48.835493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:48.836604 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:48.836609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:48.837690 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:48.837709 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:48.838831 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:48.838837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:48.838975 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:48.839581 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:48.839598 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:48.843562 (+ 3964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":266,"threads_started":1}
W20260929 11:27:48.844055 1017 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:48.834007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:40163 (local address 127.23.153.62:42209)
0929 11:27:48.834566 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:48.834570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:48.834589 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:27:48.834649 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:48.834653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:48.834718 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:48.834822 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:48.835637 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:48.836440 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:48.837875 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:48.838534 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:48.838658 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:48.843800 (+ 5142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.1:40163: BlockingRecv error: recv got EOF from 127.23.153.1:40163 (error 108)
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":300,"threads_started":1}
Sep 29 11:27:49 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:49 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:49.017115 1018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:49.006211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.3:48687)
0929 11:27:49.007223 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:49.007248 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:49.007494 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:49.007864 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:49.007869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:49.008410 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:49.008623 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:49.008635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:49.010217 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:49.010223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:49.011536 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:49.011553 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:49.012382 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:49.012387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:49.012508 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:49.013146 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:49.013166 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:49.016825 (+ 3659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":571,"threads_started":1}
W20260929 11:27:49.017352 1017 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:49.006513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:48687 (local address 127.23.153.62:42209)
0929 11:27:49.006841 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:49.006848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:49.007384 (+ 536us) server_negotiation.cc:407] Connection header received
0929 11:27:49.007658 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:49.007662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:49.007728 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:49.007807 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:49.009027 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:49.010049 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:49.011783 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:49.012215 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:49.012324 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:49.017138 (+ 4814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.3:48687: BlockingRecv error: recv got EOF from 127.23.153.3:48687 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
Sep 29 11:27:49 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:49 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:49.254555 1019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:49.244587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.2:42067)
0929 11:27:49.245163 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:49.245184 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:49.245321 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:49.245630 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:49.245634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:49.246044 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:49.246231 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:49.246240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:49.247440 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:49.247445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:49.248547 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:49.248564 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:49.249546 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:49.249552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:49.249718 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:49.250325 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:49.250363 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:49.254300 (+ 3937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":352,"threads_started":1}
W20260929 11:27:49.254814 1017 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:49.244742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:42067 (local address 127.23.153.62:42209)
0929 11:27:49.244968 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:49.244973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:49.245265 (+ 292us) server_negotiation.cc:407] Connection header received
0929 11:27:49.245417 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:49.245421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:49.245500 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:49.245600 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:49.246425 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:49.247272 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:49.248742 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:49.249311 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:49.249488 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:49.254540 (+ 5052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.2:42067: BlockingRecv error: recv got EOF from 127.23.153.2:42067 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 29 11:27:49 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:49 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:49.854063 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:49.844799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.1:34189)
0929 11:27:49.845597 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:49.845617 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:49.845749 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:49.846103 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:49.846108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:49.846616 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:49.846796 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:49.846805 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:49.847811 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:49.847815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:49.848782 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:49.848793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:49.849545 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:49.849550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:49.849678 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:49.850311 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:49.850369 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:49.853798 (+ 3429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":552,"threads_started":1}
W20260929 11:27:49.854275 1021 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:49.844872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:34189 (local address 127.23.153.62:42209)
0929 11:27:49.845652 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:49.845658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:49.845676 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:27:49.845856 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:49.845860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:49.845941 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:49.846050 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:49.846919 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:49.847706 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:49.848892 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:49.849382 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:49.849477 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:49.854045 (+ 4568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.1:34189: BlockingRecv error: recv got EOF from 127.23.153.1:34189 (error 108)
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":510,"threads_started":1}
Sep 29 11:27:50 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:50 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:50.029788 1022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:50.018564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.3:40183)
0929 11:27:50.019286 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:50.019306 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:50.019475 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:50.019826 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:50.019830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:50.020269 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:50.020459 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:50.020468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:50.021806 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:50.021812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:50.023148 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:50.023178 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:50.024031 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:50.024038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:50.024239 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:50.024901 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:50.024922 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:50.029515 (+ 4593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":483,"threads_started":1}
W20260929 11:27:50.029989 1021 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:50.018765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:40183 (local address 127.23.153.62:42209)
0929 11:27:50.019037 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:50.019044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:50.019420 (+ 376us) server_negotiation.cc:407] Connection header received
0929 11:27:50.019617 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:50.019620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:50.019690 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:50.019792 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:50.020616 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:50.021547 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:50.023366 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:50.023836 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:50.023968 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:50.029792 (+ 5824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.3:40183: BlockingRecv error: recv got EOF from 127.23.153.3:40183 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 29 11:27:50 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:50 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:50.267082 1023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:50.255576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.2:60935)
0929 11:27:50.256120 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:50.256140 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:50.256286 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:50.256648 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:50.256651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:50.257037 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:50.257208 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:50.257218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:50.258278 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:50.258284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:50.259308 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:50.259322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:50.260302 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:50.260310 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:50.260518 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:50.261563 (+ 1045us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:50.261589 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:50.266613 (+ 5024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":284,"threads_started":1}
W20260929 11:27:50.267256 1021 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:50.255814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:60935 (local address 127.23.153.62:42209)
0929 11:27:50.256013 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:50.256017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:50.256224 (+ 207us) server_negotiation.cc:407] Connection header received
0929 11:27:50.256464 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:50.256469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:50.256532 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:50.256617 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:50.257373 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:50.258127 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:50.259487 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:50.259953 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:50.260070 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:50.266963 (+ 6893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.2:60935: BlockingRecv error: recv got EOF from 127.23.153.2:60935 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 11:27:50 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:50 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:50.864687 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:50.855122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.1:49873)
0929 11:27:50.855574 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:50.855592 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:50.855704 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:50.856001 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:50.856004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:50.856386 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:50.856541 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:50.856569 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:50.857735 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:50.857739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:50.858811 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:50.858827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:50.859578 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:50.859583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:50.859728 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:50.860353 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:50.860372 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:50.864341 (+ 3969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":229,"threads_started":1}
W20260929 11:27:50.864862 1025 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:50.855209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:49873 (local address 127.23.153.62:42209)
0929 11:27:50.855684 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:50.855689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:50.855707 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:27:50.855803 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:50.855807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:50.855868 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:50.855974 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:50.856723 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:50.857588 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:50.858955 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:50.859427 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:50.859545 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:50.864603 (+ 5058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.1:49873: BlockingRecv error: recv got EOF from 127.23.153.1:49873 (error 108)
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":251,"threads_started":1}
Sep 29 11:27:51 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:51 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:51.040630 1026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:51.031066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.3:39617)
0929 11:27:51.031661 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:51.031677 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:51.031827 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:51.032166 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:51.032171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:51.032597 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:51.032773 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:51.032782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:51.034112 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:51.034117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:51.035179 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:51.035194 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:51.035998 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:51.036002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:51.036148 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:51.036745 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:51.036763 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:51.040263 (+ 3500us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":295,"threads_started":1}
W20260929 11:27:51.040753 1025 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:51.031185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:39617 (local address 127.23.153.62:42209)
0929 11:27:51.031391 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:51.031398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:51.031782 (+ 384us) server_negotiation.cc:407] Connection header received
0929 11:27:51.031950 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:51.031955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:51.032029 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:51.032114 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:51.032915 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:51.033942 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:51.035353 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:51.035813 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:51.035924 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:51.040504 (+ 4580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.3:39617: BlockingRecv error: recv got EOF from 127.23.153.3:39617 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 11:27:51 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:51 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:51.278007 1027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:51.268540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.2:52129)
0929 11:27:51.269092 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:51.269108 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:51.269248 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:51.269591 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:51.269618 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:51.270011 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:51.270177 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:51.270186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:51.271234 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:51.271238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:51.272320 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:51.272337 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:51.273223 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:51.273229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:51.273412 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:51.274016 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:51.274035 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:51.277747 (+ 3712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":357,"threads_started":1}
W20260929 11:27:51.278210 1025 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:51.268643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:52129 (local address 127.23.153.62:42209)
0929 11:27:51.268999 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:51.269006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:51.269217 (+ 211us) server_negotiation.cc:407] Connection header received
0929 11:27:51.269348 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:51.269352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:51.269433 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:51.269522 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:51.270306 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:51.271087 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:51.272480 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:51.273069 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:51.273188 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:51.277972 (+ 4784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.2:52129: BlockingRecv error: recv got EOF from 127.23.153.2:52129 (error 108)
Metrics: {"server-negotiator.queue_time_us":209}
Sep 29 11:27:51 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:51 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.1@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:51.875775 1028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:51.865972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.1:53239)
0929 11:27:51.866513 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:51.866534 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:51.866776 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:51.867204 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:51.867209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:51.867712 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:51.867911 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:51.867923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:51.869086 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:51.869092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:51.870057 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:51.870071 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:51.870839 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:51.870844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:51.870982 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:51.871663 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:51.871681 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:51.875496 (+ 3815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":263,"threads_started":1}
W20260929 11:27:51.875998 1029 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:51.866022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:53239 (local address 127.23.153.62:42209)
0929 11:27:51.866538 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:51.866543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:51.866723 (+ 180us) server_negotiation.cc:407] Connection header received
0929 11:27:51.866891 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:51.866896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:51.866974 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:51.867109 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:51.868111 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:51.868904 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:51.870231 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:51.870699 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:51.870805 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:51.875763 (+ 4958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.1:53239: BlockingRecv error: recv got EOF from 127.23.153.1:53239 (error 108)
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":252,"threads_started":1}
Sep 29 11:27:52 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:52 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.3@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:52.051940 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:52.041654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.3:51585)
0929 11:27:52.042481 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:52.042500 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:52.042616 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:52.042962 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:52.042967 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:52.043574 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:52.043737 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:52.043747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:52.044730 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:52.044734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:52.046101 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:52.046119 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:52.047179 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:52.047186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:52.047381 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:52.048246 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:52.048267 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:52.051701 (+ 3434us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":254,"threads_started":1}
W20260929 11:27:52.052336 1029 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:52.041846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:51585 (local address 127.23.153.62:42209)
0929 11:27:52.042212 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:52.042216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:52.042567 (+ 351us) server_negotiation.cc:407] Connection header received
0929 11:27:52.042737 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:52.042742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:52.042824 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:52.042910 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:52.043869 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:52.044612 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:52.046319 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:52.046972 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:52.047128 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:52.051937 (+ 4809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.3:51585: BlockingRecv error: recv got EOF from 127.23.153.3:51585 (error 108)
Metrics: {"server-negotiator.queue_time_us":177}
Sep 29 11:27:52 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 11:27:52 dist-test-slave-qk05 krb5kdc[32436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.153.2@KRBTEST.COM for oryx/127.23.153.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 11:27:52.288827 1031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:52.279411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42209 (local address 127.23.153.2:51479)
0929 11:27:52.279939 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:52.279955 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:52.280095 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:52.280456 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:52.280460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:52.280893 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:52.281079 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:52.281091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:52.282213 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:52.282217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:52.283329 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:52.283344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:52.284195 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:52.284202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:52.284365 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:52.285049 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:27:52.285068 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:52.288520 (+ 3452us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.153.62:42209: Server oryx/127.23.153.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":279,"threads_started":1}
W20260929 11:27:52.289000 1029 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:52.279603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:51479 (local address 127.23.153.62:42209)
0929 11:27:52.279850 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:52.279856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:52.280036 (+ 180us) server_negotiation.cc:407] Connection header received
0929 11:27:52.280215 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:52.280219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:52.280321 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:52.280405 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:52.281241 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:52.282049 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:52.283501 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:52.284047 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:52.284167 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:52.288779 (+ 4612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.153.2:51479: BlockingRecv error: recv got EOF from 127.23.153.2:51479 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 11:27:52.840791 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 557
I20260929 11:27:52.854552 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:52.842164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50006 (local address 127.23.153.1:39593)
0929 11:27:52.842770 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:52.842779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:52.842807 (+ 28us) server_negotiation.cc:407] Connection header received
0929 11:27:52.842874 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:52.842881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:52.843184 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:52.843349 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:52.844908 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:52.845994 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:52.847464 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:52.848072 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:52.848185 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:52.851008 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:52.851050 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:52.851071 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:52.851123 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:52.853836 (+ 2713us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.153.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 11:27:52.854326 (+ 490us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50006: Request ticket server kudu/127.23.153.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":329,"threads_started":1}
W20260929 11:27:52.854655 1033 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50006: Request ticket server kudu/127.23.153.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 11:27:52.854915 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 557: Not authorized: Client connection negotiation failed: client connection to 127.23.153.1:39593: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.153.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 11:27:52.855151 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 557
I20260929 11:27:52.870092 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 708
I20260929 11:27:52.884016 1034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:52.871493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59854 (local address 127.23.153.2:43843)
0929 11:27:52.872291 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:52.872301 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:52.872335 (+ 34us) server_negotiation.cc:407] Connection header received
0929 11:27:52.872417 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:52.872427 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:52.872789 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:52.873023 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:52.874308 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:52.875557 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:52.876966 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:52.877722 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:52.877848 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:52.880469 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:52.880480 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:52.880496 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:52.880559 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:52.883446 (+ 2887us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.153.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 11:27:52.883788 (+ 342us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59854: Request ticket server kudu/127.23.153.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":441,"threads_started":1}
W20260929 11:27:52.884136 1034 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59854: Request ticket server kudu/127.23.153.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 11:27:52.884426 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 708: Not authorized: Client connection negotiation failed: client connection to 127.23.153.2:43843: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.153.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 11:27:52.884586 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 708
I20260929 11:27:52.899039 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 841
I20260929 11:27:52.912853 1035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:52.900222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52804 (local address 127.23.153.3:40315)
0929 11:27:52.901014 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:52.901024 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:52.901056 (+ 32us) server_negotiation.cc:407] Connection header received
0929 11:27:52.901137 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:52.901146 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:52.901455 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:52.901690 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:52.903008 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:52.904221 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:52.905749 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:52.906370 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:52.906501 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:52.909207 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:52.909218 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:52.909233 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:27:52.909287 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:52.912242 (+ 2955us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.153.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 11:27:52.912598 (+ 356us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52804: Request ticket server kudu/127.23.153.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":400,"threads_started":1}
W20260929 11:27:52.912958 1035 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52804: Request ticket server kudu/127.23.153.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 11:27:52.913142 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 841: Not authorized: Client connection negotiation failed: client connection to 127.23.153.3:40315: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.153.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 11:27:52.913312 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 841
I20260929 11:27:52.929219 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 488
I20260929 11:27:52.930757 550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:53.049517 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 488
2026-09-29T11:27:53Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21688 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T11:27:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:27:53Z Disabled control of system clock
I20260929 11:27:53.128935 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:39157
--webserver_interface=127.23.153.62
--webserver_port=0
--builtin_ntp_servers=127.23.153.20:40403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:39157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 11:27:53.492177 1040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:53.492547 1040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:53.492671 1040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:53.501354 1040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:27:53.501516 1040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:53.501575 1040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:53.501655 1040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:27:53.501714 1040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:27:53.514899 1040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:40403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:39157
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:39157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:53.516477 1040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:53.518504 1040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:53.532997 1046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:53.533056 1048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:53.533267 1045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:53.533998 1040 server_base.cc:1034] running on GCE node
I20260929 11:27:53.534885 1040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:53.536616 1040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:53.538101 1040 hybrid_clock.cc:648] HybridClock initialized: now 1790681273537962 us; error 56 us; skew 500 ppm
I20260929 11:27:53.538575 1040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:27:53.546139 1040 webserver.cc:460] Webserver started at http://127.23.153.62:39371/ using document root <none> and password file <none>
I20260929 11:27:53.547046 1040 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:53.547174 1040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:53.547536 1040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:53.550992 1040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "8186488eb0484516ac1d0cc2a5b6510c"
format_stamp: "Formatted at 2026-09-29 11:27:53 on dist-test-slave-qk05"
server_key: "f1816066cd728552732ffb14fde4a8d8"
server_key_iv: "a5de298a303ec7c8ec18fd6cb7c293ae"
server_key_version: "encryptionkey@0"
I20260929 11:27:53.552488 1040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "8186488eb0484516ac1d0cc2a5b6510c"
format_stamp: "Formatted at 2026-09-29 11:27:53 on dist-test-slave-qk05"
server_key: "f1816066cd728552732ffb14fde4a8d8"
server_key_iv: "a5de298a303ec7c8ec18fd6cb7c293ae"
server_key_version: "encryptionkey@0"
I20260929 11:27:53.561488 1040 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.002s
I20260929 11:27:53.566568 1054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:53.568403 1040 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:27:53.568624 1040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "8186488eb0484516ac1d0cc2a5b6510c"
format_stamp: "Formatted at 2026-09-29 11:27:53 on dist-test-slave-qk05"
server_key: "f1816066cd728552732ffb14fde4a8d8"
server_key_iv: "a5de298a303ec7c8ec18fd6cb7c293ae"
server_key_version: "encryptionkey@0"
I20260929 11:27:53.568869 1040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:53.601222 1040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:53.602180 1040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:53.602473 1040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:53.622303 1040 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:39157
I20260929 11:27:53.622329 1105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:39157 every 8 connection(s)
I20260929 11:27:53.623771 1040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:27:53.624910 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 1040
I20260929 11:27:53.625094 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:27:53.625733 24164 external_mini_cluster.cc:1428] Setting key dbab4a4ce758af785905d13ed7ce82f2
I20260929 11:27:53.628297 1106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:53.643445 1109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:53.628022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59456 (local address 127.23.153.62:39157)
0929 11:27:53.629123 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:53.629140 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:53.629192 (+ 52us) server_negotiation.cc:407] Connection header received
0929 11:27:53.631450 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:53.631485 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:53.632384 (+ 899us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:53.632962 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:53.634034 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:53.636158 (+ 2124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:53.637701 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:53.638757 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:53.638957 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:53.640229 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:53.640242 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:53.640272 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:53.641100 (+ 828us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:53.641219 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:53.641250 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:53.641385 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:53.641647 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 11:27:53.642025 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":358,"threads_started":1}
I20260929 11:27:53.643611 1106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c: Bootstrap starting.
I20260929 11:27:53.647095 1106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:53.648414 1106 log.cc:824] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:53.652113 1106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c: No bootstrap required, opened a new log
I20260929 11:27:53.660064 1106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8186488eb0484516ac1d0cc2a5b6510c" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39157 } }
I20260929 11:27:53.660513 1106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:53.660596 1106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8186488eb0484516ac1d0cc2a5b6510c, State: Initialized, Role: FOLLOWER
I20260929 11:27:53.661164 1106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8186488eb0484516ac1d0cc2a5b6510c" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39157 } }
I20260929 11:27:53.661399 1106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:27:53.661514 1106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:27:53.661702 1106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:53.664880 1106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8186488eb0484516ac1d0cc2a5b6510c" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39157 } }
I20260929 11:27:53.665444 1106 leader_election.cc:302] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8186488eb0484516ac1d0cc2a5b6510c; no voters:
I20260929 11:27:53.666050 1106 leader_election.cc:288] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:27:53.666203 1111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:53.666911 1111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c [term 1 LEADER]: Becoming Leader. State: Replica: 8186488eb0484516ac1d0cc2a5b6510c, State: Running, Role: LEADER
I20260929 11:27:53.667511 1111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8186488eb0484516ac1d0cc2a5b6510c" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39157 } }
I20260929 11:27:53.668246 1106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:27:53.670192 1113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8186488eb0484516ac1d0cc2a5b6510c. Latest consensus state: current_term: 1 leader_uuid: "8186488eb0484516ac1d0cc2a5b6510c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8186488eb0484516ac1d0cc2a5b6510c" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39157 } } }
I20260929 11:27:53.670658 1112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8186488eb0484516ac1d0cc2a5b6510c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8186488eb0484516ac1d0cc2a5b6510c" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39157 } } }
I20260929 11:27:53.670990 1112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:53.670658 1113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:53.674348 1118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:27:53.678835 1118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:27:53.688491 1118 catalog_manager.cc:1347] Generated new cluster ID: 63890c9dfafe4e6fb41134d31fdcd19b
I20260929 11:27:53.688630 1118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:27:53.702970 1118 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:27:53.705302 1118 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:27:53.717428 1118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8186488eb0484516ac1d0cc2a5b6510c: Generated new TSK 0
I20260929 11:27:53.718385 1118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:27:53.733036 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:39157
--builtin_ntp_servers=127.23.153.20:40403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 11:27:54.126914 1130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:54.127343 1130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:54.127462 1130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:54.135488 1130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:54.135630 1130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:54.135798 1130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:27:54.146095 1130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:40403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:39157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:54.147572 1130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:54.149513 1130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:54.163818 1135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:54.163851 1136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:54.163894 1138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:54.164723 1130 server_base.cc:1034] running on GCE node
I20260929 11:27:54.165414 1130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:54.166467 1130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:54.167702 1130 hybrid_clock.cc:648] HybridClock initialized: now 1790681274167636 us; error 65 us; skew 500 ppm
I20260929 11:27:54.168075 1130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:27:54.170727 1130 webserver.cc:460] Webserver started at http://127.23.153.1:44451/ using document root <none> and password file <none>
I20260929 11:27:54.171531 1130 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:54.171648 1130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:54.171959 1130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:54.175510 1130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "8297a4137d69487f9ee4562582e67b22"
format_stamp: "Formatted at 2026-09-29 11:27:54 on dist-test-slave-qk05"
server_key: "2391860e15ffcde38195647b3704d882"
server_key_iv: "0a167f49f898b8d67bacad78888c02f3"
server_key_version: "encryptionkey@0"
I20260929 11:27:54.176437 1130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "8297a4137d69487f9ee4562582e67b22"
format_stamp: "Formatted at 2026-09-29 11:27:54 on dist-test-slave-qk05"
server_key: "2391860e15ffcde38195647b3704d882"
server_key_iv: "0a167f49f898b8d67bacad78888c02f3"
server_key_version: "encryptionkey@0"
I20260929 11:27:54.182843 1130 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 11:27:54.187537 1144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:54.189157 1130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:27:54.189388 1130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "8297a4137d69487f9ee4562582e67b22"
format_stamp: "Formatted at 2026-09-29 11:27:54 on dist-test-slave-qk05"
server_key: "2391860e15ffcde38195647b3704d882"
server_key_iv: "0a167f49f898b8d67bacad78888c02f3"
server_key_version: "encryptionkey@0"
I20260929 11:27:54.189677 1130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:54.229554 1130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:54.230552 1130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:54.230857 1130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:54.231928 1130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:54.233469 1130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:54.233569 1130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:54.233678 1130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:54.233747 1130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:54.267037 1130 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:37879
I20260929 11:27:54.267151 1256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:37879 every 8 connection(s)
I20260929 11:27:54.268487 1130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:27:54.276046 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 1130
I20260929 11:27:54.276265 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:27:54.277037 24164 external_mini_cluster.cc:1428] Setting key 09bbac243fd5e7c9abbf4e511d2ef2a8
I20260929 11:27:54.280234 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:39157
--builtin_ntp_servers=127.23.153.20:40403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 11:27:54.285849 1259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:54.272316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:43099 (local address 127.23.153.62:39157)
0929 11:27:54.272798 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:54.272803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:54.275002 (+ 2199us) server_negotiation.cc:407] Connection header received
0929 11:27:54.277704 (+ 2702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:54.277712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:54.277821 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:54.277968 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:54.279497 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:54.280382 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:54.282434 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:54.282870 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:54.282996 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:54.284894 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:54.284901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:54.284905 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:54.284928 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:54.284975 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:54.284988 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:54.285075 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:54.285342 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 11:27:54.285472 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":261,"threads_started":1}
I20260929 11:27:54.287061 1260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:54.273216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:39157 (local address 127.23.153.1:43099)
0929 11:27:54.274790 (+ 1574us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:54.274837 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:54.277247 (+ 2410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:54.278317 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:54.278328 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:54.278453 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:54.279226 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:54.279251 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:54.280642 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:54.280648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:54.282235 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:54.282256 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:54.283282 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:54.283314 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:54.283651 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:54.284442 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:27:54.284445 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:54.284495 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:27:54.284611 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:54.285094 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:54.285115 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:54.285122 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 11:27:54.285297 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 11:27:54.285805 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":459,"threads_started":1}
I20260929 11:27:54.289062 1257 heartbeater.cc:347] Connected to a master server at 127.23.153.62:39157
I20260929 11:27:54.289501 1257 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:54.290530 1257 heartbeater.cc:511] Master 127.23.153.62:39157 requested a full tablet report, sending...
I20260929 11:27:54.293495 1071 ts_manager.cc:194] Registered new tserver with Master: 8297a4137d69487f9ee4562582e67b22 (127.23.153.1:37879)
I20260929 11:27:54.296110 1071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.153.1:43099
W20260929 11:27:54.632313 1261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:54.632653 1261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:54.632795 1261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:54.640508 1261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:54.640650 1261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:54.640811 1261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:27:54.650421 1261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:40403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:39157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:54.652222 1261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:54.654209 1261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:54.668642 1267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:54.668669 1266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:54.668618 1269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:54.669482 1261 server_base.cc:1034] running on GCE node
I20260929 11:27:54.670136 1261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:54.671205 1261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:54.672446 1261 hybrid_clock.cc:648] HybridClock initialized: now 1790681274672388 us; error 48 us; skew 500 ppm
I20260929 11:27:54.672792 1261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:27:54.675318 1261 webserver.cc:460] Webserver started at http://127.23.153.2:33537/ using document root <none> and password file <none>
I20260929 11:27:54.676040 1261 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:54.676133 1261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:54.676453 1261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:54.679553 1261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "8efcae51fa274b6ba829dcfa2d21f536"
format_stamp: "Formatted at 2026-09-29 11:27:54 on dist-test-slave-qk05"
server_key: "5923bf2a87b8532cad0f19bdabdd1ef6"
server_key_iv: "6049a187f3b3836ec735e3eee56de6ca"
server_key_version: "encryptionkey@0"
I20260929 11:27:54.680509 1261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "8efcae51fa274b6ba829dcfa2d21f536"
format_stamp: "Formatted at 2026-09-29 11:27:54 on dist-test-slave-qk05"
server_key: "5923bf2a87b8532cad0f19bdabdd1ef6"
server_key_iv: "6049a187f3b3836ec735e3eee56de6ca"
server_key_version: "encryptionkey@0"
I20260929 11:27:54.686172 1261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:27:54.690266 1275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:54.692026 1261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 11:27:54.692233 1261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "8efcae51fa274b6ba829dcfa2d21f536"
format_stamp: "Formatted at 2026-09-29 11:27:54 on dist-test-slave-qk05"
server_key: "5923bf2a87b8532cad0f19bdabdd1ef6"
server_key_iv: "6049a187f3b3836ec735e3eee56de6ca"
server_key_version: "encryptionkey@0"
I20260929 11:27:54.692481 1261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:54.724198 1261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:54.725180 1261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:54.725479 1261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:54.726517 1261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:54.728098 1261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:54.728223 1261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:54.728335 1261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:54.728647 1261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:54.763067 1261 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:46259
I20260929 11:27:54.763160 1387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:46259 every 8 connection(s)
I20260929 11:27:54.764469 1261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:27:54.766840 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 1261
I20260929 11:27:54.766989 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:27:54.767513 24164 external_mini_cluster.cc:1428] Setting key 73099500ad9279068725339781f734dc
I20260929 11:27:54.770424 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:39157
--builtin_ntp_servers=127.23.153.20:40403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 11:27:54.779932 1259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:54.767326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:44127 (local address 127.23.153.62:39157)
0929 11:27:54.767548 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:54.767554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:54.769384 (+ 1830us) server_negotiation.cc:407] Connection header received
0929 11:27:54.771602 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:54.771608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:54.771701 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:54.771792 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:54.773386 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:54.774335 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:54.776325 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:54.776791 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:54.776934 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:54.778949 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:54.778956 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:54.778959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:54.778982 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:54.779042 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:54.779061 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:54.779190 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:54.779441 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 11:27:54.779561 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 11:27:54.781142 1390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:54.767999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:39157 (local address 127.23.153.2:44127)
0929 11:27:54.769228 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:54.769275 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:54.771282 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:54.772071 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:54.772081 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:54.772185 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:54.773117 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:54.773146 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:54.774535 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:54.774540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:54.776098 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:54.776122 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:54.777077 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:54.777086 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:54.777345 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:54.778507 (+ 1162us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:27:54.778510 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:54.778561 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:27:54.778675 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:54.779189 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:54.779203 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:54.779213 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 11:27:54.779420 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 11:27:54.779903 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":375,"threads_started":1}
I20260929 11:27:54.782917 1388 heartbeater.cc:347] Connected to a master server at 127.23.153.62:39157
I20260929 11:27:54.783270 1388 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:54.784107 1388 heartbeater.cc:511] Master 127.23.153.62:39157 requested a full tablet report, sending...
I20260929 11:27:54.785730 1071 ts_manager.cc:194] Registered new tserver with Master: 8efcae51fa274b6ba829dcfa2d21f536 (127.23.153.2:46259)
I20260929 11:27:54.786583 1071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.153.2:44127
W20260929 11:27:55.123945 1391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:55.124294 1391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:55.124423 1391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:55.132318 1391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:55.132453 1391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:55.132611 1391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:27:55.142989 1391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:40403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:39157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:55.144434 1391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:55.146360 1391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:55.160561 1397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:55.160530 1396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:55.160600 1399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:55.161159 1391 server_base.cc:1034] running on GCE node
I20260929 11:27:55.161835 1391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:55.162914 1391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:55.164178 1391 hybrid_clock.cc:648] HybridClock initialized: now 1790681275164112 us; error 55 us; skew 500 ppm
I20260929 11:27:55.164554 1391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:27:55.167795 1391 webserver.cc:460] Webserver started at http://127.23.153.3:40917/ using document root <none> and password file <none>
I20260929 11:27:55.168716 1391 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:55.168838 1391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:55.169170 1391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:55.172606 1391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "cb18c1144bde4b2e93f026553f3cdd6f"
format_stamp: "Formatted at 2026-09-29 11:27:55 on dist-test-slave-qk05"
server_key: "14c054ab9f8a69842dc6d32eca59594f"
server_key_iv: "3c23ee26c4a4a61bde32e76619d56978"
server_key_version: "encryptionkey@0"
I20260929 11:27:55.173734 1391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "cb18c1144bde4b2e93f026553f3cdd6f"
format_stamp: "Formatted at 2026-09-29 11:27:55 on dist-test-slave-qk05"
server_key: "14c054ab9f8a69842dc6d32eca59594f"
server_key_iv: "3c23ee26c4a4a61bde32e76619d56978"
server_key_version: "encryptionkey@0"
I20260929 11:27:55.180081 1391 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 11:27:55.184297 1405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:55.186020 1391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:27:55.186236 1391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "cb18c1144bde4b2e93f026553f3cdd6f"
format_stamp: "Formatted at 2026-09-29 11:27:55 on dist-test-slave-qk05"
server_key: "14c054ab9f8a69842dc6d32eca59594f"
server_key_iv: "3c23ee26c4a4a61bde32e76619d56978"
server_key_version: "encryptionkey@0"
I20260929 11:27:55.186479 1391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:55.214519 1391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:55.215489 1391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:55.215796 1391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:55.216779 1391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:55.218313 1391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:55.218418 1391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:55.218551 1391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:55.218623 1391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:55.250687 1391 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:39901
I20260929 11:27:55.250747 1517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:39901 every 8 connection(s)
I20260929 11:27:55.252404 1391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:27:55.257349 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 1391
I20260929 11:27:55.257522 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:27:55.258081 24164 external_mini_cluster.cc:1428] Setting key 3eea7e81b5a043ae07ecf904e0737365
I20260929 11:27:55.267879 1259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:55.255882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:56407 (local address 127.23.153.62:39157)
0929 11:27:55.256093 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:55.256098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:55.258254 (+ 2156us) server_negotiation.cc:407] Connection header received
0929 11:27:55.260454 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:55.260459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:55.260677 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:55.260900 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:55.262223 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:55.262999 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:55.264848 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:55.265279 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:55.265402 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:55.266977 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:55.266985 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:55.266990 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:55.267052 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:55.267094 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:55.267112 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:55.267209 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:55.267424 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 11:27:55.267537 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 11:27:55.268888 1520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:55.256780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:39157 (local address 127.23.153.3:56407)
0929 11:27:55.258043 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:55.258102 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:55.260171 (+ 2069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:55.261160 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:55.261171 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:55.261283 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:55.262032 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:55.262054 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:55.263192 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:55.263197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:55.264671 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:27:55.264690 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:27:55.265539 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:27:55.265546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:27:55.265784 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:55.266583 (+ 799us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:27:55.266587 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:55.266643 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:27:55.266744 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:55.267207 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:55.267214 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:55.267221 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 11:27:55.267388 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 11:27:55.267745 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":411,"threads_started":1}
I20260929 11:27:55.270462 1518 heartbeater.cc:347] Connected to a master server at 127.23.153.62:39157
I20260929 11:27:55.270829 1518 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:55.271917 1518 heartbeater.cc:511] Master 127.23.153.62:39157 requested a full tablet report, sending...
I20260929 11:27:55.274314 1071 ts_manager.cc:194] Registered new tserver with Master: cb18c1144bde4b2e93f026553f3cdd6f (127.23.153.3:39901)
I20260929 11:27:55.275228 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:27:55.275434 1071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.153.3:56407
W20260929 11:27:55.281637 1259 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:55.280115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.23.153.62:39157)
0929 11:27:55.280306 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:55.280311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:55.280776 (+ 465us) server_negotiation.cc:407] Connection header received
0929 11:27:55.280922 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:55.280926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:55.280993 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:55.281092 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:55.281425 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59472: BlockingRecv error: recv got EOF from 127.0.0.1:59472 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:27:55.282368 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 1130
I20260929 11:27:55.290699 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:55.283397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34536 (local address 127.23.153.1:37879)
0929 11:27:55.283885 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:55.283900 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:55.283931 (+ 31us) server_negotiation.cc:407] Connection header received
0929 11:27:55.284007 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:55.284013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:55.284248 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:55.284398 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:55.284766 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:55.285816 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:55.287336 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:55.288014 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:55.288129 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:55.289207 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:55.289215 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:55.289229 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:55.289820 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:55.289874 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:55.289895 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:55.289988 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:55.290185 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 11:27:55.290310 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":237,"threads_started":1}
I20260929 11:27:55.292106 1252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:55.374197 1257 heartbeater.cc:503] Master 127.23.153.62:39157 was elected leader, sending a full tablet report...
I20260929 11:27:55.375247 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 1130
W20260929 11:27:55.389393 1056 connection.cc:535] server connection from 127.23.153.1:43099 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 11:27:55.389797 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 1261
I20260929 11:27:55.400754 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:55.390977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.23.153.2:46259)
0929 11:27:55.391747 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:55.391755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:55.391885 (+ 130us) server_negotiation.cc:407] Connection header received
0929 11:27:55.392023 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:55.392031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:55.392378 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:55.392617 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:55.393262 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:55.394458 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:55.396038 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:55.396994 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:55.397141 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:55.398524 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:55.398535 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:55.398551 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:55.399690 (+ 1139us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:55.399762 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:55.399810 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:55.399939 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:55.400164 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 11:27:55.400324 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":393,"threads_started":1}
I20260929 11:27:55.402573 1383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:55.487725 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 1261
I20260929 11:27:55.502010 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 1391
I20260929 11:27:55.510977 1531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:55.503057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48474 (local address 127.23.153.3:39901)
0929 11:27:55.503829 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:55.503838 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:55.503872 (+ 34us) server_negotiation.cc:407] Connection header received
0929 11:27:55.503934 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:55.503941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:55.504190 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:55.504329 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:55.504825 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:55.505909 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:55.507314 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:27:55.507978 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:27:55.508094 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:27:55.509301 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:55.509310 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:55.509326 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:55.510100 (+ 774us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:55.510158 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:55.510197 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:55.510298 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:55.510504 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 11:27:55.510635 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":345,"threads_started":1}
I20260929 11:27:55.512420 1513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:55.602170 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 1391
I20260929 11:27:55.615475 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 1040
I20260929 11:27:55.616801 1101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:55.722972 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 1040
2026-09-29T11:27:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2668 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T11:27:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:27:55Z Disabled control of system clock
I20260929 11:27:55.792897 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:44929
--webserver_interface=127.23.153.62
--webserver_port=0
--builtin_ntp_servers=127.23.153.20:40859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:44929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 11:27:56.159999 1537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:56.160318 1537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:56.160404 1537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:56.168356 1537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:27:56.168504 1537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:56.168541 1537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:56.168597 1537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:27:56.168642 1537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:27:56.178588 1537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:40859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:44929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:44929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:56.179944 1537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:56.181787 1537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:56.194209 1542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:56.194145 1543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:56.194183 1545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:56.195005 1537 server_base.cc:1034] running on GCE node
I20260929 11:27:56.195871 1537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:56.197435 1537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:56.198664 1537 hybrid_clock.cc:648] HybridClock initialized: now 1790681276198618 us; error 40 us; skew 500 ppm
I20260929 11:27:56.199081 1537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:27:56.201514 1537 webserver.cc:460] Webserver started at http://127.23.153.62:41405/ using document root <none> and password file <none>
I20260929 11:27:56.202275 1537 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:56.202389 1537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:56.202723 1537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:56.207463 1537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "981fabe108c445818cfc9dacd21d4fa1"
format_stamp: "Formatted at 2026-09-29 11:27:56 on dist-test-slave-qk05"
server_key: "d04e98a9dd63f37fc19c5d8655dc091b"
server_key_iv: "a646fc4b9dc6af9c0ec088c6d0c138e4"
server_key_version: "encryptionkey@0"
I20260929 11:27:56.208376 1537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "981fabe108c445818cfc9dacd21d4fa1"
format_stamp: "Formatted at 2026-09-29 11:27:56 on dist-test-slave-qk05"
server_key: "d04e98a9dd63f37fc19c5d8655dc091b"
server_key_iv: "a646fc4b9dc6af9c0ec088c6d0c138e4"
server_key_version: "encryptionkey@0"
I20260929 11:27:56.214367 1537 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260929 11:27:56.218483 1551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:56.220376 1537 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:27:56.220592 1537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "981fabe108c445818cfc9dacd21d4fa1"
format_stamp: "Formatted at 2026-09-29 11:27:56 on dist-test-slave-qk05"
server_key: "d04e98a9dd63f37fc19c5d8655dc091b"
server_key_iv: "a646fc4b9dc6af9c0ec088c6d0c138e4"
server_key_version: "encryptionkey@0"
I20260929 11:27:56.220836 1537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:56.227149 1537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:56.228062 1537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:56.228370 1537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:56.247836 1537 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:44929
I20260929 11:27:56.247851 1602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:44929 every 8 connection(s)
I20260929 11:27:56.249295 1537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:27:56.253449 1603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:27:56.257185 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 1537
I20260929 11:27:56.257400 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:27:56.258013 24164 external_mini_cluster.cc:1428] Setting key fa64b283f749d955ebb677ac7ff62331
I20260929 11:27:56.267524 1603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1: Bootstrap starting.
I20260929 11:27:56.267695 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:56.260476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.23.153.62:44929)
0929 11:27:56.261645 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:56.261658 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:56.261716 (+ 58us) server_negotiation.cc:407] Connection header received
0929 11:27:56.263519 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:56.263548 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:56.264118 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:56.264584 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:56.265789 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:56.265797 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:56.265814 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:56.265830 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:56.265898 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:56.265914 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:56.266145 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:56.266288 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 11:27:56.266600 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":408,"threads_started":1}
I20260929 11:27:56.271052 1603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1: Neither blocks nor log segments found. Creating new log.
I20260929 11:27:56.272212 1603 log.cc:824] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1: Log is configured to *not* fsync() on all Append() calls
I20260929 11:27:56.276196 1603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1: No bootstrap required, opened a new log
I20260929 11:27:56.284861 1603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981fabe108c445818cfc9dacd21d4fa1" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44929 } }
I20260929 11:27:56.285271 1603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:27:56.285372 1603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 981fabe108c445818cfc9dacd21d4fa1, State: Initialized, Role: FOLLOWER
I20260929 11:27:56.286026 1603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981fabe108c445818cfc9dacd21d4fa1" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44929 } }
I20260929 11:27:56.286254 1603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:27:56.286383 1603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:27:56.286551 1603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:27:56.288906 1603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981fabe108c445818cfc9dacd21d4fa1" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44929 } }
I20260929 11:27:56.289606 1603 leader_election.cc:302] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 981fabe108c445818cfc9dacd21d4fa1; no voters:
I20260929 11:27:56.290302 1603 leader_election.cc:288] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:27:56.290436 1608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:27:56.291143 1608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1 [term 1 LEADER]: Becoming Leader. State: Replica: 981fabe108c445818cfc9dacd21d4fa1, State: Running, Role: LEADER
I20260929 11:27:56.291766 1608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981fabe108c445818cfc9dacd21d4fa1" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44929 } }
I20260929 11:27:56.292443 1603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:27:56.294726 1609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "981fabe108c445818cfc9dacd21d4fa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981fabe108c445818cfc9dacd21d4fa1" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44929 } } }
I20260929 11:27:56.295308 1609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:56.295238 1610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 981fabe108c445818cfc9dacd21d4fa1. Latest consensus state: current_term: 1 leader_uuid: "981fabe108c445818cfc9dacd21d4fa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981fabe108c445818cfc9dacd21d4fa1" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44929 } } }
I20260929 11:27:56.295553 1610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1 [sys.catalog]: This master's current role is: LEADER
I20260929 11:27:56.296445 1616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:27:56.302373 1616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:27:56.313144 1616 catalog_manager.cc:1347] Generated new cluster ID: b7c5f5ddf4af47e1b9c66790bc46f7ce
I20260929 11:27:56.313282 1616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:27:56.338796 1616 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:27:56.339373 1616 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:27:56.352901 1616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 981fabe108c445818cfc9dacd21d4fa1: Generated new TSK 0
I20260929 11:27:56.353837 1616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:27:56.367880 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:44929
--builtin_ntp_servers=127.23.153.20:40859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 11:27:56.738524 1627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:56.738857 1627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:56.738977 1627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:56.748338 1627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:56.748481 1627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:56.748637 1627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:27:56.759378 1627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:40859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:44929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:56.760830 1627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:56.762852 1627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:56.778059 1635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:56.778139 1633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:56.778108 1632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:56.780939 1627 server_base.cc:1034] running on GCE node
I20260929 11:27:56.781739 1627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:56.782888 1627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:56.784235 1627 hybrid_clock.cc:648] HybridClock initialized: now 1790681276784169 us; error 44 us; skew 500 ppm
I20260929 11:27:56.784799 1627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:27:56.787732 1627 webserver.cc:460] Webserver started at http://127.23.153.1:38857/ using document root <none> and password file <none>
I20260929 11:27:56.788528 1627 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:56.788642 1627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:56.788971 1627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:56.792300 1627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "e04e4241e16c45f79b50d4f340f65598"
format_stamp: "Formatted at 2026-09-29 11:27:56 on dist-test-slave-qk05"
server_key: "eb453a580b2db371f17d368d6801f485"
server_key_iv: "1ea452a8f116f81008ba8423ff1502b1"
server_key_version: "encryptionkey@0"
I20260929 11:27:56.793318 1627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "e04e4241e16c45f79b50d4f340f65598"
format_stamp: "Formatted at 2026-09-29 11:27:56 on dist-test-slave-qk05"
server_key: "eb453a580b2db371f17d368d6801f485"
server_key_iv: "1ea452a8f116f81008ba8423ff1502b1"
server_key_version: "encryptionkey@0"
I20260929 11:27:56.802686 1627 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.004s
I20260929 11:27:56.807188 1641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:56.808794 1627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:27:56.808987 1627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "e04e4241e16c45f79b50d4f340f65598"
format_stamp: "Formatted at 2026-09-29 11:27:56 on dist-test-slave-qk05"
server_key: "eb453a580b2db371f17d368d6801f485"
server_key_iv: "1ea452a8f116f81008ba8423ff1502b1"
server_key_version: "encryptionkey@0"
I20260929 11:27:56.809245 1627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:56.816216 1627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:56.817178 1627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:56.817718 1627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:56.819422 1627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:56.821658 1627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:56.821785 1627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:56.821892 1627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:56.821961 1627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:56.856855 1627 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:38281
I20260929 11:27:56.856902 1753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:38281 every 8 connection(s)
I20260929 11:27:56.858248 1627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:27:56.865603 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 1627
I20260929 11:27:56.865849 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:27:56.866456 24164 external_mini_cluster.cc:1428] Setting key c16f10722107995bdb571ca7422bdeaf
I20260929 11:27:56.868371 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:56.861067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:52903 (local address 127.23.153.62:44929)
0929 11:27:56.861544 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:56.861550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:56.863814 (+ 2264us) server_negotiation.cc:407] Connection header received
0929 11:27:56.865810 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:56.865816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:56.865886 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:56.866017 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:56.867563 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:56.867569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:56.867574 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:56.867593 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:56.867638 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:56.867651 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:56.867765 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:56.867975 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 11:27:56.868054 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":252,"threads_started":1}
I20260929 11:27:56.869264 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:44929
--builtin_ntp_servers=127.23.153.20:40859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 11:27:56.869500 1757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:56.862143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44929 (local address 127.23.153.1:52903)
0929 11:27:56.863595 (+ 1452us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:56.863640 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:56.865392 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:56.866326 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:56.866337 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:56.866454 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:56.867219 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:27:56.867222 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:56.867269 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:27:56.867371 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:56.867757 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:56.867768 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:56.867776 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 11:27:56.868069 (+ 293us) client_negotiation.cc:240] Negotiation successful
0929 11:27:56.868401 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":443,"threads_started":1}
I20260929 11:27:56.871434 1754 heartbeater.cc:347] Connected to a master server at 127.23.153.62:44929
I20260929 11:27:56.871778 1754 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:56.872428 1754 heartbeater.cc:511] Master 127.23.153.62:44929 requested a full tablet report, sending...
I20260929 11:27:56.874962 1568 ts_manager.cc:194] Registered new tserver with Master: e04e4241e16c45f79b50d4f340f65598 (127.23.153.1:38281)
W20260929 11:27:57.226930 1758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:57.227308 1758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:57.227414 1758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:57.235687 1758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:57.235831 1758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:57.235956 1758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:27:57.247074 1758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:40859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:44929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:57.248586 1758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:57.250478 1758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:57.271338 1766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:57.271343 1764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:57.272148 1763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:57.272923 1758 server_base.cc:1034] running on GCE node
I20260929 11:27:57.273628 1758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:57.274691 1758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:57.275919 1758 hybrid_clock.cc:648] HybridClock initialized: now 1790681277275864 us; error 54 us; skew 500 ppm
I20260929 11:27:57.276265 1758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:27:57.278741 1758 webserver.cc:460] Webserver started at http://127.23.153.2:45247/ using document root <none> and password file <none>
I20260929 11:27:57.279618 1758 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:57.279735 1758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:57.280082 1758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:57.283315 1758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "9ef1366037744465b4da97863458aca9"
format_stamp: "Formatted at 2026-09-29 11:27:57 on dist-test-slave-qk05"
server_key: "885a26e98a41e5892fda27331b0a75bb"
server_key_iv: "459f5f24a82c9752bf3801b474a504b8"
server_key_version: "encryptionkey@0"
I20260929 11:27:57.284430 1758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "9ef1366037744465b4da97863458aca9"
format_stamp: "Formatted at 2026-09-29 11:27:57 on dist-test-slave-qk05"
server_key: "885a26e98a41e5892fda27331b0a75bb"
server_key_iv: "459f5f24a82c9752bf3801b474a504b8"
server_key_version: "encryptionkey@0"
I20260929 11:27:57.290733 1758 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 11:27:57.295195 1772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:57.296865 1758 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:27:57.297056 1758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "9ef1366037744465b4da97863458aca9"
format_stamp: "Formatted at 2026-09-29 11:27:57 on dist-test-slave-qk05"
server_key: "885a26e98a41e5892fda27331b0a75bb"
server_key_iv: "459f5f24a82c9752bf3801b474a504b8"
server_key_version: "encryptionkey@0"
I20260929 11:27:57.297293 1758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:57.303880 1758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:57.304804 1758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:57.305094 1758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:57.306082 1758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:57.307660 1758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:57.307768 1758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:57.307891 1758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:57.307961 1758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:57.339514 1758 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:34019
I20260929 11:27:57.339581 1884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:34019 every 8 connection(s)
I20260929 11:27:57.340888 1758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:27:57.347942 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 1758
I20260929 11:27:57.348117 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:27:57.348665 24164 external_mini_cluster.cc:1428] Setting key a2700cc3a06bcfa305f00d1931205f91
I20260929 11:27:57.349485 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:57.343472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:33459 (local address 127.23.153.62:44929)
0929 11:27:57.343664 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:57.343668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:57.345230 (+ 1562us) server_negotiation.cc:407] Connection header received
0929 11:27:57.346965 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:57.346970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:57.347077 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:57.347170 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:57.348644 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:57.348650 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:57.348653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:57.348671 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:57.348702 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:57.348716 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:57.348849 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:57.349116 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 11:27:57.349186 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 11:27:57.350584 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:57.344055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44929 (local address 127.23.153.2:33459)
0929 11:27:57.345044 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:57.345083 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:57.346655 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:57.347433 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:57.347443 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:57.347549 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:57.348264 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:27:57.348267 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:57.348322 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:27:57.348449 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:57.348910 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:57.348918 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:57.348925 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 11:27:57.349191 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 11:27:57.349470 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":301,"threads_started":1}
I20260929 11:27:57.351743 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:44929
--builtin_ntp_servers=127.23.153.20:40859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 11:27:57.352267 1885 heartbeater.cc:347] Connected to a master server at 127.23.153.62:44929
I20260929 11:27:57.352587 1885 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:57.353302 1885 heartbeater.cc:511] Master 127.23.153.62:44929 requested a full tablet report, sending...
I20260929 11:27:57.354912 1568 ts_manager.cc:194] Registered new tserver with Master: 9ef1366037744465b4da97863458aca9 (127.23.153.2:34019)
W20260929 11:27:57.706619 1888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:57.706972 1888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:57.707146 1888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:57.716104 1888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:57.716231 1888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:57.716391 1888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:27:57.726312 1888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:40859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:44929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:57.727785 1888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:57.729689 1888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:57.745224 1894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:57.745270 1896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:57.745306 1893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:57.746044 1888 server_base.cc:1034] running on GCE node
I20260929 11:27:57.746729 1888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:57.747879 1888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:57.749104 1888 hybrid_clock.cc:648] HybridClock initialized: now 1790681277749041 us; error 66 us; skew 500 ppm
I20260929 11:27:57.749492 1888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:27:57.752134 1888 webserver.cc:460] Webserver started at http://127.23.153.3:43745/ using document root <none> and password file <none>
I20260929 11:27:57.752888 1888 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:57.753000 1888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:57.753333 1888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:57.756860 1888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "f0c73dd7d37945358b03fcfa50479c00"
format_stamp: "Formatted at 2026-09-29 11:27:57 on dist-test-slave-qk05"
server_key: "69ca2eeeea613aca403a93e872a2923e"
server_key_iv: "0eeea884406a26498b6e89adcb1d5eb0"
server_key_version: "encryptionkey@0"
I20260929 11:27:57.757771 1888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "f0c73dd7d37945358b03fcfa50479c00"
format_stamp: "Formatted at 2026-09-29 11:27:57 on dist-test-slave-qk05"
server_key: "69ca2eeeea613aca403a93e872a2923e"
server_key_iv: "0eeea884406a26498b6e89adcb1d5eb0"
server_key_version: "encryptionkey@0"
I20260929 11:27:57.763854 1888 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 11:27:57.768088 1902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:57.769608 1888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260929 11:27:57.769804 1888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "f0c73dd7d37945358b03fcfa50479c00"
format_stamp: "Formatted at 2026-09-29 11:27:57 on dist-test-slave-qk05"
server_key: "69ca2eeeea613aca403a93e872a2923e"
server_key_iv: "0eeea884406a26498b6e89adcb1d5eb0"
server_key_version: "encryptionkey@0"
I20260929 11:27:57.770040 1888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:27:57.776381 1888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:27:57.777298 1888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:27:57.777578 1888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:27:57.778576 1888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:27:57.780263 1888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:27:57.780380 1888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:57.780511 1888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:27:57.780597 1888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:57.815236 1888 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:38509
I20260929 11:27:57.815277 2014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:38509 every 8 connection(s)
I20260929 11:27:57.816644 1888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:27:57.826025 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 1888
I20260929 11:27:57.826224 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:27:57.826292 1756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:57.819913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:42977 (local address 127.23.153.62:44929)
0929 11:27:57.820192 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:57.820199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:57.821785 (+ 1586us) server_negotiation.cc:407] Connection header received
0929 11:27:57.823674 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:57.823680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:57.823743 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:57.823854 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:57.825451 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:57.825456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:57.825460 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:57.825480 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:57.825515 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:57.825528 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:57.825676 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:57.825871 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 11:27:57.825944 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260929 11:27:57.826726 24164 external_mini_cluster.cc:1428] Setting key 43e004c4c04b10e06a10b9c25888b814
I20260929 11:27:57.827404 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:57.820499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44929 (local address 127.23.153.3:42977)
0929 11:27:57.821580 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0929 11:27:57.821625 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 11:27:57.823307 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:27:57.824161 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:27:57.824172 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:27:57.824285 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:27:57.825021 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:27:57.825024 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:27:57.825093 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:27:57.825223 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:27:57.825659 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:27:57.825669 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:27:57.825676 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 11:27:57.825927 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 11:27:57.826202 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":335,"threads_started":1}
I20260929 11:27:57.828893 2015 heartbeater.cc:347] Connected to a master server at 127.23.153.62:44929
I20260929 11:27:57.829260 2015 heartbeater.cc:464] Registering TS with master...
I20260929 11:27:57.830014 2015 heartbeater.cc:511] Master 127.23.153.62:44929 requested a full tablet report, sending...
I20260929 11:27:57.831676 1568 ts_manager.cc:194] Registered new tserver with Master: f0c73dd7d37945358b03fcfa50479c00 (127.23.153.3:38509)
I20260929 11:27:57.842885 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 11:27:57.849217 1756 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:27:57.847663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.23.153.62:44929)
0929 11:27:57.847837 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:57.847844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:57.848296 (+ 452us) server_negotiation.cc:407] Connection header received
0929 11:27:57.848476 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:57.848480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:57.848533 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:57.848620 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:57.848992 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36876: BlockingRecv error: recv got EOF from 127.0.0.1:36876 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:27:57.850142 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 1627
I20260929 11:27:57.853900 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:57.851247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46630 (local address 127.23.153.1:38281)
0929 11:27:57.851748 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:57.851755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:57.851785 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:27:57.851855 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:57.851862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:57.852073 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:57.852200 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:57.853079 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:57.853086 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:57.853100 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:57.853119 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:57.853159 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:57.853173 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:57.853283 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:57.853342 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 11:27:57.853428 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":239,"threads_started":1}
I20260929 11:27:57.855288 1749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:57.940546 1754 heartbeater.cc:503] Master 127.23.153.62:44929 was elected leader, sending a full tablet report...
I20260929 11:27:57.941708 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 1627
W20260929 11:27:57.956210 1554 connection.cc:535] server connection from 127.23.153.1:52903 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 11:27:57.956517 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 1758
I20260929 11:27:57.960690 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:57.957874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55356 (local address 127.23.153.2:34019)
0929 11:27:57.958396 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:57.958402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:57.958430 (+ 28us) server_negotiation.cc:407] Connection header received
0929 11:27:57.958492 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:57.958498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:57.958814 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:57.958962 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:57.960039 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:57.960047 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:57.960059 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:57.960072 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:57.960121 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:57.960136 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:57.960245 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:57.960299 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 11:27:57.960381 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":243,"threads_started":1}
I20260929 11:27:57.962013 1880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:58.045861 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 1758
I20260929 11:27:58.059173 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 1888
I20260929 11:27:58.062880 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:27:58.060045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40240 (local address 127.23.153.3:38509)
0929 11:27:58.060595 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0929 11:27:58.060602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:27:58.060656 (+ 54us) server_negotiation.cc:407] Connection header received
0929 11:27:58.060718 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:27:58.060724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:27:58.060953 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:27:58.061096 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:27:58.062146 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:27:58.062152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:27:58.062164 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:27:58.062176 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:27:58.062218 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:27:58.062235 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:27:58.062397 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 11:27:58.062466 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 11:27:58.062573 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":262,"threads_started":1}
I20260929 11:27:58.064280 2010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:58.151486 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 1888
I20260929 11:27:58.165586 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 1537
I20260929 11:27:58.166875 1598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:27:58.269642 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 1537
2026-09-29T11:27:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2547 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:27:58 dist-test-slave-qk05 krb5kdc[2033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:27:58 dist-test-slave-qk05 krb5kdc[2033](info): set up 2 sockets
Sep 29 11:27:58 dist-test-slave-qk05 krb5kdc[2033](info): commencing operation
krb5kdc: starting...
W20260929 11:27:59.324012 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:27:59 dist-test-slave-qk05 krb5kdc[2033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:27:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:27:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:27:59.571921 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:38435
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:38435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:27:59.929513 2046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:27:59.929860 2046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:27:59.929977 2046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:27:59.937523 2046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:27:59.937651 2046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:27:59.937701 2046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:27:59.937784 2046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:27:59.937839 2046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:27:59.948194 2046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:38435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:38435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:27:59.949741 2046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:27:59.951187 2046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:27:59.963081 2051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:59.963078 2052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:27:59.963599 2054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:27:59.963840 2046 server_base.cc:1034] running on GCE node
I20260929 11:27:59.964655 2046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:27:59.966176 2046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:27:59.967402 2046 hybrid_clock.cc:648] HybridClock initialized: now 1790681279967340 us; error 58 us; skew 500 ppm
I20260929 11:27:59.967775 2046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:27:59 dist-test-slave-qk05 krb5kdc[2033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681279, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:27:59.975724 2046 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:27:59.977527 2046 webserver.cc:460] Webserver started at http://127.23.153.62:40813/ using document root <none> and password file <none>
I20260929 11:27:59.978247 2046 fs_manager.cc:359] Metadata directory not provided
I20260929 11:27:59.978363 2046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:27:59.978638 2046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:27:59.981846 2046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "cc6ac44d2e3349f981c6f3c1abd02d98"
format_stamp: "Formatted at 2026-09-29 11:27:59 on dist-test-slave-qk05"
server_key: "81cc5826a7f8e1cf40ed44cb1a065956"
server_key_iv: "489bea5aaca493dc126cdd90ab92e708"
server_key_version: "encryptionkey@0"
I20260929 11:27:59.982616 2046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "cc6ac44d2e3349f981c6f3c1abd02d98"
format_stamp: "Formatted at 2026-09-29 11:27:59 on dist-test-slave-qk05"
server_key: "81cc5826a7f8e1cf40ed44cb1a065956"
server_key_iv: "489bea5aaca493dc126cdd90ab92e708"
server_key_version: "encryptionkey@0"
I20260929 11:27:59.989168 2046 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260929 11:27:59.993604 2061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:27:59.995980 2046 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 11:27:59.996210 2046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "cc6ac44d2e3349f981c6f3c1abd02d98"
format_stamp: "Formatted at 2026-09-29 11:27:59 on dist-test-slave-qk05"
server_key: "81cc5826a7f8e1cf40ed44cb1a065956"
server_key_iv: "489bea5aaca493dc126cdd90ab92e708"
server_key_version: "encryptionkey@0"
I20260929 11:27:59.996415 2046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:28:00.018309 2046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:28:00.023381 2046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:28:00.023699 2046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:28:00.043160 2046 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:38435
I20260929 11:28:00.043183 2112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:38435 every 8 connection(s)
I20260929 11:28:00.044646 2046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:28:00.046543 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 2046
I20260929 11:28:00.046787 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:28:00.047794 24164 external_mini_cluster.cc:1428] Setting key abe6720c8dd2cbe56ac76ee1302c737c
I20260929 11:28:00.049989 2113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 11:28:00 dist-test-slave-qk05 krb5kdc[2033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:28:00.065652 2113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98: Bootstrap starting.
I20260929 11:28:00.069334 2113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98: Neither blocks nor log segments found. Creating new log.
I20260929 11:28:00.070700 2113 log.cc:824] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98: Log is configured to *not* fsync() on all Append() calls
I20260929 11:28:00.074601 2113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98: No bootstrap required, opened a new log
I20260929 11:28:00.074723 2116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:00.050726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38214 (local address 127.23.153.62:38435)
0929 11:28:00.051867 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:00.051880 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:00.051927 (+ 47us) server_negotiation.cc:407] Connection header received
0929 11:28:00.053543 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:00.053569 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:00.054057 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:00.054466 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:28:00.056477 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:00.057980 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:00.059770 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:00.060551 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:00.060737 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:00.066096 (+ 5359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:28:00.066107 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:28:00.066128 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:28:00.066172 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:28:00.069463 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:00.070342 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:00.070351 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:00.070360 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:00.070639 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:00.071135 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:00.071140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:00.071143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:00.071984 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:28:00.072284 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:00.072832 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 11:28:00.073326 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878,"thread_start_us":584,"threads_started":1}
I20260929 11:28:00.085065 2113 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6ac44d2e3349f981c6f3c1abd02d98" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38435 } }
I20260929 11:28:00.085640 2113 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:28:00.085757 2113 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc6ac44d2e3349f981c6f3c1abd02d98, State: Initialized, Role: FOLLOWER
I20260929 11:28:00.086359 2113 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6ac44d2e3349f981c6f3c1abd02d98" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38435 } }
I20260929 11:28:00.086591 2113 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:28:00.086699 2113 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:28:00.086985 2113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:28:00.089478 2113 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6ac44d2e3349f981c6f3c1abd02d98" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38435 } }
I20260929 11:28:00.090031 2113 leader_election.cc:302] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc6ac44d2e3349f981c6f3c1abd02d98; no voters:
I20260929 11:28:00.090688 2113 leader_election.cc:288] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:28:00.090845 2118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:28:00.091579 2118 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98 [term 1 LEADER]: Becoming Leader. State: Replica: cc6ac44d2e3349f981c6f3c1abd02d98, State: Running, Role: LEADER
I20260929 11:28:00.092219 2118 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6ac44d2e3349f981c6f3c1abd02d98" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38435 } }
I20260929 11:28:00.092897 2113 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:28:00.095100 2119 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc6ac44d2e3349f981c6f3c1abd02d98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6ac44d2e3349f981c6f3c1abd02d98" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38435 } } }
I20260929 11:28:00.095530 2119 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98 [sys.catalog]: This master's current role is: LEADER
I20260929 11:28:00.096120 2120 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc6ac44d2e3349f981c6f3c1abd02d98. Latest consensus state: current_term: 1 leader_uuid: "cc6ac44d2e3349f981c6f3c1abd02d98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6ac44d2e3349f981c6f3c1abd02d98" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38435 } } }
I20260929 11:28:00.097075 2120 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98 [sys.catalog]: This master's current role is: LEADER
I20260929 11:28:00.097880 2127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:28:00.102721 2127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:28:00.111692 2127 catalog_manager.cc:1347] Generated new cluster ID: ab8099ce1f174215aa933bbb26a2b251
I20260929 11:28:00.111820 2127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:28:00.139370 2127 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:28:00.141289 2127 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:28:00.151335 2127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98: Generated new TSK 0
I20260929 11:28:00.152558 2127 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
I20260929 11:28:00.214617 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38435
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:28:00.577090 2139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:28:00.577445 2139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:28:00.577589 2139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:28:00.586974 2139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:28:00.587399 2139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:28:00.587662 2139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:28:00.599617 2139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:28:00.601186 2139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:28:00.602666 2139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:28:00.617558 2145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:28:00.617650 2144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:28:00.617635 2147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:28:00.618465 2139 server_base.cc:1034] running on GCE node
I20260929 11:28:00.619194 2139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:28:00.620373 2139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:28:00.621646 2139 hybrid_clock.cc:648] HybridClock initialized: now 1790681280621531 us; error 100 us; skew 500 ppm
I20260929 11:28:00.621994 2139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:28:00 dist-test-slave-qk05 krb5kdc[2033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681280, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:28:00.625469 2139 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:28:00.627327 2139 webserver.cc:460] Webserver started at http://127.23.153.1:45155/ using document root <none> and password file <none>
I20260929 11:28:00.628115 2139 fs_manager.cc:359] Metadata directory not provided
I20260929 11:28:00.628252 2139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:28:00.628609 2139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:28:00.632431 2139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "188a084ea3b34612a66714986c05c6e9"
format_stamp: "Formatted at 2026-09-29 11:28:00 on dist-test-slave-qk05"
server_key: "4bf24b24e185db7eb0c8e3345e41a3b3"
server_key_iv: "007bc27f783cfab6fd21bfb02a22719e"
server_key_version: "encryptionkey@0"
I20260929 11:28:00.633322 2139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "188a084ea3b34612a66714986c05c6e9"
format_stamp: "Formatted at 2026-09-29 11:28:00 on dist-test-slave-qk05"
server_key: "4bf24b24e185db7eb0c8e3345e41a3b3"
server_key_iv: "007bc27f783cfab6fd21bfb02a22719e"
server_key_version: "encryptionkey@0"
I20260929 11:28:00.640162 2139 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 11:28:00.644625 2154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:28:00.646402 2139 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:28:00.646622 2139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "188a084ea3b34612a66714986c05c6e9"
format_stamp: "Formatted at 2026-09-29 11:28:00 on dist-test-slave-qk05"
server_key: "4bf24b24e185db7eb0c8e3345e41a3b3"
server_key_iv: "007bc27f783cfab6fd21bfb02a22719e"
server_key_version: "encryptionkey@0"
I20260929 11:28:00.646850 2139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:28:00.670817 2139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:28:00.675477 2139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:28:00.675796 2139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:28:00.677070 2139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:28:00.679049 2139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:28:00.679177 2139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 11:28:00.679317 2139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:28:00.679373 2139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:28:00.712582 2139 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:45363
I20260929 11:28:00.712623 2266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:45363 every 8 connection(s)
I20260929 11:28:00.713908 2139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:28:00.721935 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 2139
I20260929 11:28:00.722153 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:28:00.722754 24164 external_mini_cluster.cc:1428] Setting key 61d8610ecbaff1549ae2c91e746b8999
Sep 29 11:28:00 dist-test-slave-qk05 krb5kdc[2033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681280, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:28:00.735694 2269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:00.716598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:46533 (local address 127.23.153.62:38435)
0929 11:28:00.717064 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:00.717069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:00.718079 (+ 1010us) server_negotiation.cc:407] Connection header received
0929 11:28:00.720102 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:00.720109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:00.720214 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:00.720311 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:28:00.722006 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:00.722789 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:00.724643 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:00.725053 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:00.725177 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:00.729201 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:28:00.729207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:28:00.729210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:28:00.729249 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:28:00.732068 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:00.732963 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:00.732968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:00.732970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:00.733107 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:00.733705 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:00.733709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:00.733711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:00.734019 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:28:00.734182 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:00.735050 (+ 868us) server_negotiation.cc:299] Negotiation successful
0929 11:28:00.735336 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":248,"threads_started":1}
I20260929 11:28:00.736704 2270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:00.717103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:38435 (local address 127.23.153.1:46533)
0929 11:28:00.717878 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0929 11:28:00.717920 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 11:28:00.719757 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:28:00.720610 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:28:00.720619 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:28:00.721181 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:28:00.721796 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:00.721820 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:00.722954 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:00.722958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:00.724481 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:00.724496 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:00.725300 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:00.725308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:00.725534 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:00.726241 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:28:00.726265 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:28:00.728897 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:28:00.732280 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:28:00.732292 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:28:00.732311 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:28:00.732817 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:28:00.733311 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:28:00.733317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:28:00.733321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:28:00.733555 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:28:00.734169 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:28:00.734178 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:28:00.734585 (+ 407us) client_negotiation.cc:769] Sending connection context
0929 11:28:00.734849 (+ 264us) client_negotiation.cc:240] Negotiation successful
0929 11:28:00.735486 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":298,"threads_started":1}
I20260929 11:28:00.738322 2267 heartbeater.cc:347] Connected to a master server at 127.23.153.62:38435
I20260929 11:28:00.738696 2267 heartbeater.cc:464] Registering TS with master...
I20260929 11:28:00.739635 2267 heartbeater.cc:511] Master 127.23.153.62:38435 requested a full tablet report, sending...
I20260929 11:28:00.742410 2078 ts_manager.cc:194] Registered new tserver with Master: 188a084ea3b34612a66714986c05c6e9 (127.23.153.1:45363)
I20260929 11:28:00.744000 2078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:46533
WARNING: no policy specified for kudu/127.23.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
I20260929 11:28:00.775089 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38435
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:28:01.152284 2273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:28:01.152638 2273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:28:01.152787 2273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:28:01.160941 2273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:28:01.161092 2273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:28:01.161255 2273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:28:01.171623 2273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:28:01.173017 2273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:28:01.174397 2273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:28:01.189590 2278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:28:01.189685 2281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:28:01.189664 2279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:28:01.190312 2273 server_base.cc:1034] running on GCE node
I20260929 11:28:01.191047 2273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:28:01.192114 2273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:28:01.193346 2273 hybrid_clock.cc:648] HybridClock initialized: now 1790681281193281 us; error 59 us; skew 500 ppm
I20260929 11:28:01.193693 2273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:28:01 dist-test-slave-qk05 krb5kdc[2033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681281, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:28:01.197393 2273 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:28:01.199162 2273 webserver.cc:460] Webserver started at http://127.23.153.2:38307/ using document root <none> and password file <none>
I20260929 11:28:01.199965 2273 fs_manager.cc:359] Metadata directory not provided
I20260929 11:28:01.200088 2273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:28:01.200464 2273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:28:01.203747 2273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "9cd0f35be2d2445dad070cb7d292fea7"
format_stamp: "Formatted at 2026-09-29 11:28:01 on dist-test-slave-qk05"
server_key: "cdcb0d36ce57771801d65809d99831e8"
server_key_iv: "e209d44c8b0ea7d0a97d75f4b44c7b94"
server_key_version: "encryptionkey@0"
I20260929 11:28:01.204591 2273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "9cd0f35be2d2445dad070cb7d292fea7"
format_stamp: "Formatted at 2026-09-29 11:28:01 on dist-test-slave-qk05"
server_key: "cdcb0d36ce57771801d65809d99831e8"
server_key_iv: "e209d44c8b0ea7d0a97d75f4b44c7b94"
server_key_version: "encryptionkey@0"
I20260929 11:28:01.210700 2273 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 11:28:01.214931 2288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:28:01.216742 2273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:28:01.217057 2273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "9cd0f35be2d2445dad070cb7d292fea7"
format_stamp: "Formatted at 2026-09-29 11:28:01 on dist-test-slave-qk05"
server_key: "cdcb0d36ce57771801d65809d99831e8"
server_key_iv: "e209d44c8b0ea7d0a97d75f4b44c7b94"
server_key_version: "encryptionkey@0"
I20260929 11:28:01.217306 2273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:28:01.269848 2273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:28:01.274550 2273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:28:01.275081 2273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:28:01.276415 2273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:28:01.278337 2273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:28:01.278456 2273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:28:01.278584 2273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:28:01.278652 2273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:28:01.316154 2400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:45079 every 8 connection(s)
I20260929 11:28:01.316242 2273 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:45079
I20260929 11:28:01.318372 2273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:28:01.321142 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 2273
I20260929 11:28:01.321367 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:28:01.321841 24164 external_mini_cluster.cc:1428] Setting key e7e1271ce47d5d322bfc7223f3b21bc2
Sep 29 11:28:01 dist-test-slave-qk05 krb5kdc[2033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681281, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:28:01.342626 2403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:01.321405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:45075 (local address 127.23.153.62:38435)
0929 11:28:01.322064 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:01.322071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:01.323330 (+ 1259us) server_negotiation.cc:407] Connection header received
0929 11:28:01.325654 (+ 2324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:01.325660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:01.325755 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:01.325907 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:28:01.327750 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.328514 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.330222 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.330654 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.330774 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:01.335153 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:28:01.335158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:28:01.335162 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:28:01.335201 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:28:01.338202 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:01.339397 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:01.339403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:01.339405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:01.339565 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:01.340093 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:01.340097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:01.340099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:01.340397 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:28:01.340542 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:01.341968 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0929 11:28:01.342250 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":248,"threads_started":1}
I20260929 11:28:01.343756 2404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:01.322187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:38435 (local address 127.23.153.2:45075)
0929 11:28:01.323168 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0929 11:28:01.323219 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 11:28:01.325329 (+ 2110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:28:01.326236 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:28:01.326246 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:28:01.326816 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:28:01.327534 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:01.327561 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.328705 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.328709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:01.330035 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:01.330052 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.330905 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.330913 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:01.331200 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:01.331935 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:28:01.331959 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:28:01.334728 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:28:01.338423 (+ 3695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:28:01.338433 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:28:01.338450 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:28:01.339228 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:28:01.339717 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:28:01.339721 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:28:01.339724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:28:01.339963 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:28:01.340535 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:28:01.340546 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:28:01.341304 (+ 758us) client_negotiation.cc:769] Sending connection context
0929 11:28:01.341789 (+ 485us) client_negotiation.cc:240] Negotiation successful
0929 11:28:01.342467 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":368,"threads_started":1}
WARNING: no policy specified for kudu/127.23.153.3@KRBTEST.COM; defaulting to no policy
I20260929 11:28:01.345391 2401 heartbeater.cc:347] Connected to a master server at 127.23.153.62:38435
I20260929 11:28:01.345849 2401 heartbeater.cc:464] Registering TS with master...
I20260929 11:28:01.346923 2401 heartbeater.cc:511] Master 127.23.153.62:38435 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.3@KRBTEST.COM" created.
I20260929 11:28:01.348685 2078 ts_manager.cc:194] Registered new tserver with Master: 9cd0f35be2d2445dad070cb7d292fea7 (127.23.153.2:45079)
I20260929 11:28:01.349566 2078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:45075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
I20260929 11:28:01.379386 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38435
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:28:01.747514 2267 heartbeater.cc:503] Master 127.23.153.62:38435 was elected leader, sending a full tablet report...
W20260929 11:28:01.757848 2407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:28:01.758211 2407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:28:01.758347 2407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:28:01.766476 2407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:28:01.766631 2407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:28:01.766773 2407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:28:01.777292 2407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:28:01.778684 2407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:28:01.780120 2407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:28:01.794518 2412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:28:01.794561 2413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:28:01.794587 2415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:28:01.795598 2407 server_base.cc:1034] running on GCE node
I20260929 11:28:01.796279 2407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:28:01.797374 2407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:28:01.798650 2407 hybrid_clock.cc:648] HybridClock initialized: now 1790681281798593 us; error 50 us; skew 500 ppm
I20260929 11:28:01.799001 2407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:28:01 dist-test-slave-qk05 krb5kdc[2033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681281, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:28:01.803371 2407 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:28:01.805013 2407 webserver.cc:460] Webserver started at http://127.23.153.3:44513/ using document root <none> and password file <none>
I20260929 11:28:01.805962 2407 fs_manager.cc:359] Metadata directory not provided
I20260929 11:28:01.806108 2407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:28:01.806456 2407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:28:01.809800 2407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "974efdb1a834448696a7990d8ff930a6"
format_stamp: "Formatted at 2026-09-29 11:28:01 on dist-test-slave-qk05"
server_key: "5bbe6e16ef638427b006bf002e4bb1d1"
server_key_iv: "0ec2b9e5328524bca4402068dbfe8c67"
server_key_version: "encryptionkey@0"
I20260929 11:28:01.810668 2407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "974efdb1a834448696a7990d8ff930a6"
format_stamp: "Formatted at 2026-09-29 11:28:01 on dist-test-slave-qk05"
server_key: "5bbe6e16ef638427b006bf002e4bb1d1"
server_key_iv: "0ec2b9e5328524bca4402068dbfe8c67"
server_key_version: "encryptionkey@0"
I20260929 11:28:01.816560 2407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 11:28:01.820691 2422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:28:01.822359 2407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:28:01.822628 2407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "974efdb1a834448696a7990d8ff930a6"
format_stamp: "Formatted at 2026-09-29 11:28:01 on dist-test-slave-qk05"
server_key: "5bbe6e16ef638427b006bf002e4bb1d1"
server_key_iv: "0ec2b9e5328524bca4402068dbfe8c67"
server_key_version: "encryptionkey@0"
I20260929 11:28:01.822839 2407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:28:01.838160 2407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:28:01.844030 2407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:28:01.844350 2407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:28:01.845474 2407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:28:01.846982 2407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:28:01.847107 2407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 11:28:01.847317 2407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:28:01.847400 2407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:28:01.881989 2407 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:36643
I20260929 11:28:01.882042 2534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:36643 every 8 connection(s)
I20260929 11:28:01.883489 2407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:28:01.883909 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 2407
I20260929 11:28:01.884091 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:28:01.884604 24164 external_mini_cluster.cc:1428] Setting key 7194443cc549ae0d9a2c952a04619bfb
Sep 29 11:28:01 dist-test-slave-qk05 krb5kdc[2033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681281, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:28:01.906375 2537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:01.886507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:55655 (local address 127.23.153.62:38435)
0929 11:28:01.886966 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:01.886972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:01.888494 (+ 1522us) server_negotiation.cc:407] Connection header received
0929 11:28:01.890568 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:01.890573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:01.890654 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:01.890760 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:28:01.893191 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.893993 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.895910 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.896503 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.896614 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:01.900541 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:28:01.900546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:28:01.900550 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:28:01.900590 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:28:01.903271 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:01.904080 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:01.904084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:01.904087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:01.904206 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:01.904691 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:01.904696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:01.904698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:01.904995 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:28:01.905137 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:01.905893 (+ 756us) server_negotiation.cc:299] Negotiation successful
0929 11:28:01.906087 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":229,"threads_started":1}
I20260929 11:28:01.907496 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:01.887299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:38435 (local address 127.23.153.3:55655)
0929 11:28:01.888320 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0929 11:28:01.888377 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 11:28:01.890092 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:28:01.891212 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:28:01.891225 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:28:01.891805 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:28:01.892929 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:01.892975 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.894238 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.894244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:01.895719 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:01.895737 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.896746 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.896752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:01.897006 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:01.897720 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:28:01.897746 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:28:01.900308 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:28:01.903454 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:28:01.903463 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:28:01.903480 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:28:01.903964 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:28:01.904295 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:28:01.904299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:28:01.904302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:28:01.904537 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:28:01.905171 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:28:01.905178 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:28:01.905549 (+ 371us) client_negotiation.cc:769] Sending connection context
0929 11:28:01.905803 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 11:28:01.906233 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":511,"threads_started":1}
I20260929 11:28:01.909027 2535 heartbeater.cc:347] Connected to a master server at 127.23.153.62:38435
I20260929 11:28:01.909355 2535 heartbeater.cc:464] Registering TS with master...
I20260929 11:28:01.910122 2535 heartbeater.cc:511] Master 127.23.153.62:38435 requested a full tablet report, sending...
I20260929 11:28:01.911712 2078 ts_manager.cc:194] Registered new tserver with Master: 974efdb1a834448696a7990d8ff930a6 (127.23.153.3:36643)
I20260929 11:28:01.913043 2078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:55655
I20260929 11:28:01.913347 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:28:01.931691 2537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:01.917839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.23.153.62:38435)
0929 11:28:01.918052 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:01.918127 (+ 75us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:01.918398 (+ 271us) server_negotiation.cc:407] Connection header received
0929 11:28:01.918538 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:01.918542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:01.918621 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:01.918724 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:28:01.920183 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.920978 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.922431 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.922864 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.922983 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:01.925472 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:28:01.925478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:28:01.925482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:28:01.925524 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:28:01.928471 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:01.929507 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:01.929512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:01.929515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:01.929665 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:01.930169 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:01.930173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:01.930175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:01.930531 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:28:01.930738 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:01.931199 (+ 461us) server_negotiation.cc:299] Negotiation successful
0929 11:28:01.931385 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:28:01.936270 2078 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:28:01.943637 2078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:28:01.983747 2550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:01.958406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:45363 (local address 127.0.0.1:58934)
0929 11:28:01.973045 (+ 14639us) negotiation.cc:107] Waiting for socket to connect
0929 11:28:01.973060 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:28:01.973183 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:28:01.974237 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:28:01.974245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:28:01.974302 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:28:01.975003 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:01.975043 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.977253 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.977260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:01.980390 (+ 3130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:01.980400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.982567 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.982575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:01.982818 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:01.982897 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 11:28:01.983045 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 11:28:01.983276 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14482,"mutex_wait_us":220,"spinlock_wait_cycles":6016,"thread_start_us":14325,"threads_started":1}
I20260929 11:28:01.983740 2549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:01.958194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50462 (local address 127.23.153.3:36643)
0929 11:28:01.971660 (+ 13466us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:01.971672 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:01.973097 (+ 1425us) server_negotiation.cc:407] Connection header received
0929 11:28:01.973493 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:01.973501 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:01.973756 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:01.973942 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:28:01.975304 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.976948 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.980574 (+ 3626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.982095 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.982223 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:01.982462 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:01.983235 (+ 773us) server_negotiation.cc:299] Negotiation successful
0929 11:28:01.983402 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13218,"thread_start_us":13505,"threads_started":1}
I20260929 11:28:01.984108 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:01.958576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58934 (local address 127.23.153.1:45363)
0929 11:28:01.972585 (+ 14009us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:01.972594 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:01.973395 (+ 801us) server_negotiation.cc:407] Connection header received
0929 11:28:01.973491 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:01.973499 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:01.973756 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:01.973970 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:28:01.975317 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.976947 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.980589 (+ 3642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.982311 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.982445 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:01.982658 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:01.983131 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 11:28:01.983298 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13828,"thread_start_us":13668,"threads_started":1}
I20260929 11:28:01.984463 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:01.958003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:36643 (local address 127.0.0.1:50462)
0929 11:28:01.972853 (+ 14850us) negotiation.cc:107] Waiting for socket to connect
0929 11:28:01.972913 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 11:28:01.973183 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:28:01.974237 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:28:01.974245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:28:01.974300 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:28:01.974956 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:01.974972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.977270 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.977274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:01.980347 (+ 3073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:01.980365 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.982611 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.982618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:01.982895 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:01.982935 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 11:28:01.983054 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 11:28:01.984061 (+ 1007us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14509,"mutex_wait_us":1059,"spinlock_wait_cycles":9344,"thread_start_us":15131,"threads_started":1}
I20260929 11:28:01.987360 2553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:01.974666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:45079 (local address 127.0.0.1:56076)
0929 11:28:01.978644 (+ 3978us) negotiation.cc:107] Waiting for socket to connect
0929 11:28:01.978663 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:28:01.978774 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:28:01.979661 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:28:01.979666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:28:01.979693 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:28:01.980112 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:01.980122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.981933 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.981939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:01.984227 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:01.984241 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.986349 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.986356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:01.986541 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:01.986589 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 11:28:01.986689 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 11:28:01.986837 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3857,"mutex_wait_us":28,"thread_start_us":3668,"threads_started":1}
I20260929 11:28:01.988214 2552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:01.973936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56076 (local address 127.23.153.2:45079)
0929 11:28:01.979107 (+ 5171us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:01.979115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:01.979142 (+ 27us) server_negotiation.cc:407] Connection header received
0929 11:28:01.979222 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:01.979228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:01.979453 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:01.979645 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:28:01.980274 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.981758 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.984390 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:01.985738 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:01.985876 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:01.986098 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:01.987649 (+ 1551us) server_negotiation.cc:299] Negotiation successful
0929 11:28:01.987869 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4978,"thread_start_us":4383,"threads_started":1}
I20260929 11:28:01.989025 2202 tablet_service.cc:1467] Processing CreateTablet for tablet cac3186de8134d108abfe17b6b99129c (DEFAULT_TABLE table=test-table [id=1f876120d30b474384fdb50e46a18630]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:28:01.989030 2470 tablet_service.cc:1467] Processing CreateTablet for tablet cac3186de8134d108abfe17b6b99129c (DEFAULT_TABLE table=test-table [id=1f876120d30b474384fdb50e46a18630]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:28:01.990434 2202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cac3186de8134d108abfe17b6b99129c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:28:01.990464 2470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cac3186de8134d108abfe17b6b99129c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:28:01.992830 2336 tablet_service.cc:1467] Processing CreateTablet for tablet cac3186de8134d108abfe17b6b99129c (DEFAULT_TABLE table=test-table [id=1f876120d30b474384fdb50e46a18630]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:28:01.994549 2336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cac3186de8134d108abfe17b6b99129c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:28:02.005395 2554 tablet_bootstrap.cc:492] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9: Bootstrap starting.
I20260929 11:28:02.008278 2555 tablet_bootstrap.cc:492] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6: Bootstrap starting.
I20260929 11:28:02.010893 2554 tablet_bootstrap.cc:654] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9: Neither blocks nor log segments found. Creating new log.
I20260929 11:28:02.012133 2556 tablet_bootstrap.cc:492] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7: Bootstrap starting.
I20260929 11:28:02.012315 2555 tablet_bootstrap.cc:654] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6: Neither blocks nor log segments found. Creating new log.
I20260929 11:28:02.012375 2554 log.cc:824] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9: Log is configured to *not* fsync() on all Append() calls
I20260929 11:28:02.013857 2555 log.cc:824] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6: Log is configured to *not* fsync() on all Append() calls
I20260929 11:28:02.016182 2556 tablet_bootstrap.cc:654] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7: Neither blocks nor log segments found. Creating new log.
I20260929 11:28:02.017709 2555 tablet_bootstrap.cc:492] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6: No bootstrap required, opened a new log
I20260929 11:28:02.017707 2556 log.cc:824] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7: Log is configured to *not* fsync() on all Append() calls
I20260929 11:28:02.018033 2555 ts_tablet_manager.cc:1397] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 11:28:02.019918 2554 tablet_bootstrap.cc:492] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9: No bootstrap required, opened a new log
I20260929 11:28:02.020226 2554 ts_tablet_manager.cc:1397] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260929 11:28:02.026796 2556 tablet_bootstrap.cc:492] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7: No bootstrap required, opened a new log
I20260929 11:28:02.027652 2556 ts_tablet_manager.cc:1397] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260929 11:28:02.028964 2554 raft_consensus.cc:348] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974efdb1a834448696a7990d8ff930a6" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36643 } } peers { permanent_uuid: "188a084ea3b34612a66714986c05c6e9" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45363 } } peers { permanent_uuid: "9cd0f35be2d2445dad070cb7d292fea7" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45079 } }
I20260929 11:28:02.029408 2554 raft_consensus.cc:374] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:28:02.029806 2554 raft_consensus.cc:729] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 188a084ea3b34612a66714986c05c6e9, State: Initialized, Role: FOLLOWER
I20260929 11:28:02.029633 2555 raft_consensus.cc:348] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974efdb1a834448696a7990d8ff930a6" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36643 } } peers { permanent_uuid: "188a084ea3b34612a66714986c05c6e9" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45363 } } peers { permanent_uuid: "9cd0f35be2d2445dad070cb7d292fea7" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45079 } }
I20260929 11:28:02.030102 2555 raft_consensus.cc:374] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:28:02.030263 2555 raft_consensus.cc:729] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974efdb1a834448696a7990d8ff930a6, State: Initialized, Role: FOLLOWER
I20260929 11:28:02.030527 2554 consensus_queue.cc:260] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974efdb1a834448696a7990d8ff930a6" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36643 } } peers { permanent_uuid: "188a084ea3b34612a66714986c05c6e9" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45363 } } peers { permanent_uuid: "9cd0f35be2d2445dad070cb7d292fea7" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45079 } }
I20260929 11:28:02.030870 2555 consensus_queue.cc:260] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974efdb1a834448696a7990d8ff930a6" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36643 } } peers { permanent_uuid: "188a084ea3b34612a66714986c05c6e9" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45363 } } peers { permanent_uuid: "9cd0f35be2d2445dad070cb7d292fea7" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45079 } }
I20260929 11:28:02.032702 2554 ts_tablet_manager.cc:1428] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260929 11:28:02.034102 2535 heartbeater.cc:503] Master 127.23.153.62:38435 was elected leader, sending a full tablet report...
I20260929 11:28:02.034548 2555 ts_tablet_manager.cc:1428] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260929 11:28:02.038488 2556 raft_consensus.cc:348] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974efdb1a834448696a7990d8ff930a6" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36643 } } peers { permanent_uuid: "188a084ea3b34612a66714986c05c6e9" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45363 } } peers { permanent_uuid: "9cd0f35be2d2445dad070cb7d292fea7" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45079 } }
I20260929 11:28:02.038931 2556 raft_consensus.cc:374] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:28:02.039048 2556 raft_consensus.cc:729] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd0f35be2d2445dad070cb7d292fea7, State: Initialized, Role: FOLLOWER
I20260929 11:28:02.039634 2556 consensus_queue.cc:260] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974efdb1a834448696a7990d8ff930a6" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36643 } } peers { permanent_uuid: "188a084ea3b34612a66714986c05c6e9" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45363 } } peers { permanent_uuid: "9cd0f35be2d2445dad070cb7d292fea7" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45079 } }
I20260929 11:28:02.040788 2401 heartbeater.cc:503] Master 127.23.153.62:38435 was elected leader, sending a full tablet report...
I20260929 11:28:02.041389 2556 ts_tablet_manager.cc:1428] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260929 11:28:02.070974 2402 tablet.cc:2369] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:28:02.130451 2560 raft_consensus.cc:482] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:28:02.130780 2560 raft_consensus.cc:504] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974efdb1a834448696a7990d8ff930a6" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36643 } } peers { permanent_uuid: "188a084ea3b34612a66714986c05c6e9" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45363 } } peers { permanent_uuid: "9cd0f35be2d2445dad070cb7d292fea7" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45079 } }
I20260929 11:28:02.132754 2560 leader_election.cc:288] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 974efdb1a834448696a7990d8ff930a6 (127.23.153.3:36643), 9cd0f35be2d2445dad070cb7d292fea7 (127.23.153.2:45079)
W20260929 11:28:02.135227 2536 tablet.cc:2369] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:28:02.141502 2552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:02.133301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:47879 (local address 127.23.153.2:45079)
0929 11:28:02.133574 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:02.133580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:02.133848 (+ 268us) server_negotiation.cc:407] Connection header received
0929 11:28:02.134166 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:02.134171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:02.134245 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:02.134309 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:28:02.135097 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:02.136173 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:02.138823 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:02.140269 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:02.140462 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:02.140543 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:02.141091 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 11:28:02.141249 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260929 11:28:02.141502 2549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:02.133198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:55567 (local address 127.23.153.3:36643)
0929 11:28:02.133415 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:02.133421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:02.134342 (+ 921us) server_negotiation.cc:407] Connection header received
0929 11:28:02.134632 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:02.134636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:02.134696 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:02.134783 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:28:02.135456 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:02.136464 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:02.138823 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:02.140269 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:02.140467 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:02.140543 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:02.141091 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 11:28:02.141229 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:28:02.141685 2564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:02.133036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:36643 (local address 127.23.153.1:55567)
0929 11:28:02.134239 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0929 11:28:02.134254 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 11:28:02.134455 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:28:02.134814 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:28:02.134818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:28:02.134845 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:28:02.135271 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:02.135284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:02.136593 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:02.136602 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:02.138623 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:02.138633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:02.140598 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:02.140609 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:02.140871 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:02.140941 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 11:28:02.141056 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 11:28:02.141225 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1099,"mutex_wait_us":391,"thread_start_us":508,"threads_started":1}
I20260929 11:28:02.142071 2563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:02.133036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:45079 (local address 127.23.153.1:47879)
0929 11:28:02.133727 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 11:28:02.133759 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 11:28:02.133979 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:28:02.134515 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:28:02.134519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:28:02.134545 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:28:02.134920 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:02.134934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:02.136311 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:02.136315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:02.138555 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:02.138583 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:02.140627 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:02.140632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:02.140865 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:02.140941 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 11:28:02.141056 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 11:28:02.141225 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":74,"thread_start_us":522,"threads_started":1}
I20260929 11:28:02.142781 2490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cac3186de8134d108abfe17b6b99129c" candidate_uuid: "188a084ea3b34612a66714986c05c6e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974efdb1a834448696a7990d8ff930a6" is_pre_election: true
I20260929 11:28:02.143085 2356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cac3186de8134d108abfe17b6b99129c" candidate_uuid: "188a084ea3b34612a66714986c05c6e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cd0f35be2d2445dad070cb7d292fea7" is_pre_election: true
I20260929 11:28:02.143709 2490 raft_consensus.cc:2454] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 188a084ea3b34612a66714986c05c6e9 in term 0.
I20260929 11:28:02.143913 2356 raft_consensus.cc:2454] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 188a084ea3b34612a66714986c05c6e9 in term 0.
I20260929 11:28:02.144619 2158 leader_election.cc:302] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 188a084ea3b34612a66714986c05c6e9, 9cd0f35be2d2445dad070cb7d292fea7; no voters:
I20260929 11:28:02.144997 2560 raft_consensus.cc:2781] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:28:02.145134 2560 raft_consensus.cc:482] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:28:02.145254 2560 raft_consensus.cc:3037] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:28:02.147680 2560 raft_consensus.cc:504] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974efdb1a834448696a7990d8ff930a6" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36643 } } peers { permanent_uuid: "188a084ea3b34612a66714986c05c6e9" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45363 } } peers { permanent_uuid: "9cd0f35be2d2445dad070cb7d292fea7" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45079 } }
I20260929 11:28:02.148381 2560 leader_election.cc:288] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [CANDIDATE]: Term 1 election: Requested vote from peers 974efdb1a834448696a7990d8ff930a6 (127.23.153.3:36643), 9cd0f35be2d2445dad070cb7d292fea7 (127.23.153.2:45079)
I20260929 11:28:02.148959 2490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cac3186de8134d108abfe17b6b99129c" candidate_uuid: "188a084ea3b34612a66714986c05c6e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "974efdb1a834448696a7990d8ff930a6"
I20260929 11:28:02.149017 2356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cac3186de8134d108abfe17b6b99129c" candidate_uuid: "188a084ea3b34612a66714986c05c6e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cd0f35be2d2445dad070cb7d292fea7"
I20260929 11:28:02.149215 2490 raft_consensus.cc:3037] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:28:02.149214 2356 raft_consensus.cc:3037] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:28:02.152535 2356 raft_consensus.cc:2454] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 188a084ea3b34612a66714986c05c6e9 in term 1.
I20260929 11:28:02.152535 2490 raft_consensus.cc:2454] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 188a084ea3b34612a66714986c05c6e9 in term 1.
I20260929 11:28:02.153123 2156 leader_election.cc:302] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 188a084ea3b34612a66714986c05c6e9, 974efdb1a834448696a7990d8ff930a6, 9cd0f35be2d2445dad070cb7d292fea7; no voters:
I20260929 11:28:02.153507 2560 raft_consensus.cc:2781] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:28:02.153961 2560 raft_consensus.cc:686] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [term 1 LEADER]: Becoming Leader. State: Replica: 188a084ea3b34612a66714986c05c6e9, State: Running, Role: LEADER
I20260929 11:28:02.154462 2560 consensus_queue.cc:237] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974efdb1a834448696a7990d8ff930a6" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36643 } } peers { permanent_uuid: "188a084ea3b34612a66714986c05c6e9" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45363 } } peers { permanent_uuid: "9cd0f35be2d2445dad070cb7d292fea7" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45079 } }
I20260929 11:28:02.159880 2077 catalog_manager.cc:5495] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 188a084ea3b34612a66714986c05c6e9 (127.23.153.1). New cstate: current_term: 1 leader_uuid: "188a084ea3b34612a66714986c05c6e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974efdb1a834448696a7990d8ff930a6" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "188a084ea3b34612a66714986c05c6e9" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cd0f35be2d2445dad070cb7d292fea7" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45079 } health_report { overall_health: UNKNOWN } } }
I20260929 11:28:02.185070 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:02.179213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58938 (local address 127.23.153.1:45363)
0929 11:28:02.179412 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:02.179417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:02.179437 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:28:02.179555 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:02.179559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:02.179637 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:02.179778 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:28:02.180144 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:02.180903 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:02.182309 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:02.182754 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:02.182866 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:02.183804 (+ 938us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:28:02.184379 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:28:02.184538 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:02.184712 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 11:28:02.184804 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:28:02.193881 2490 raft_consensus.cc:1261] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6 [term 1 FOLLOWER]: Refusing update from remote peer 188a084ea3b34612a66714986c05c6e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:28:02.193881 2356 raft_consensus.cc:1261] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7 [term 1 FOLLOWER]: Refusing update from remote peer 188a084ea3b34612a66714986c05c6e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:28:02.195163 2565 consensus_queue.cc:1035] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "974efdb1a834448696a7990d8ff930a6" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:28:02.195712 2560 consensus_queue.cc:1035] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cd0f35be2d2445dad070cb7d292fea7" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:28:02.206431 2568 mvcc.cc:205] Tried to move back new op lower bound from 7334630531855200256 to 7334630531708870656. Current Snapshot: MvccSnapshot[applied={T|T < 7334630531855200256}]
I20260929 11:28:02.213584 2569 mvcc.cc:205] Tried to move back new op lower bound from 7334630531855200256 to 7334630531708870656. Current Snapshot: MvccSnapshot[applied={T|T < 7334630531855200256}]
I20260929 11:28:02.223693 2077 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:28:02.224081 2077 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:28:02.228533 2077 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc6ac44d2e3349f981c6f3c1abd02d98: Sending DeleteTablet for 3 replicas of tablet cac3186de8134d108abfe17b6b99129c
I20260929 11:28:02.229696 2202 tablet_service.cc:1514] Processing DeleteTablet for tablet cac3186de8134d108abfe17b6b99129c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:28:02 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:58934
I20260929 11:28:02.229882 2470 tablet_service.cc:1514] Processing DeleteTablet for tablet cac3186de8134d108abfe17b6b99129c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:28:02 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:50462
I20260929 11:28:02.229899 2469 tablet_service.cc:1514] Processing DeleteTablet for tablet cac3186de8134d108abfe17b6b99129c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:28:02 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:50462
I20260929 11:28:02.230803 2580 tablet_replica.cc:330] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6: stopping tablet replica
I20260929 11:28:02.230777 2336 tablet_service.cc:1514] Processing DeleteTablet for tablet cac3186de8134d108abfe17b6b99129c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:28:02 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:56076
I20260929 11:28:02.231235 2580 raft_consensus.cc:2229] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:28:02.231679 2580 raft_consensus.cc:2258] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:28:02.233394 2580 ts_tablet_manager.cc:1905] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:28:02.238893 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 2139
I20260929 11:28:02.239667 2580 ts_tablet_manager.cc:1918] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:28:02.239850 2580 log.cc:1196] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/cac3186de8134d108abfe17b6b99129c
I20260929 11:28:02.240351 2580 ts_tablet_manager.cc:1939] T cac3186de8134d108abfe17b6b99129c P 974efdb1a834448696a7990d8ff930a6: Deleting consensus metadata
I20260929 11:28:02.243134 2063 catalog_manager.cc:4841] TS 974efdb1a834448696a7990d8ff930a6 (127.23.153.3:36643): tablet cac3186de8134d108abfe17b6b99129c (table test-table [id=1f876120d30b474384fdb50e46a18630]) successfully deleted
W20260929 11:28:02.243628 2063 catalog_manager.cc:4808] TS 974efdb1a834448696a7990d8ff930a6 (127.23.153.3:36643): delete failed for tablet cac3186de8134d108abfe17b6b99129c because the tablet was not found. No further retry: Not found: Tablet not found: cac3186de8134d108abfe17b6b99129c
I20260929 11:28:02.249815 2581 tablet_replica.cc:330] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7: stopping tablet replica
I20260929 11:28:02.250571 2579 tablet_replica.cc:330] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9: stopping tablet replica
I20260929 11:28:02.251173 2579 raft_consensus.cc:2229] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:28:02.252252 2581 raft_consensus.cc:2229] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:28:02.252678 2579 raft_consensus.cc:2258] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:28:02.252841 2581 raft_consensus.cc:2258] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:28:02.255441 2581 ts_tablet_manager.cc:1905] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:28:02.255451 2579 ts_tablet_manager.cc:1905] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:28:02.262766 2581 ts_tablet_manager.cc:1918] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:28:02.262971 2581 log.cc:1196] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/cac3186de8134d108abfe17b6b99129c
I20260929 11:28:02.263345 2579 ts_tablet_manager.cc:1918] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:28:02.263530 2579 log.cc:1196] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/cac3186de8134d108abfe17b6b99129c
I20260929 11:28:02.263566 2581 ts_tablet_manager.cc:1939] T cac3186de8134d108abfe17b6b99129c P 9cd0f35be2d2445dad070cb7d292fea7: Deleting consensus metadata
I20260929 11:28:02.264027 2579 ts_tablet_manager.cc:1939] T cac3186de8134d108abfe17b6b99129c P 188a084ea3b34612a66714986c05c6e9: Deleting consensus metadata
Sep 29 11:28:02 dist-test-slave-qk05 krb5kdc[2033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:28:02.266573 2065 catalog_manager.cc:4841] TS 9cd0f35be2d2445dad070cb7d292fea7 (127.23.153.2:45079): tablet cac3186de8134d108abfe17b6b99129c (table test-table [id=1f876120d30b474384fdb50e46a18630]) successfully deleted
I20260929 11:28:02.267189 2065 catalog_manager.cc:4841] TS 188a084ea3b34612a66714986c05c6e9 (127.23.153.1:45363): tablet cac3186de8134d108abfe17b6b99129c (table test-table [id=1f876120d30b474384fdb50e46a18630]) successfully deleted
I20260929 11:28:02.276804 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:02.242695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.23.153.1:45363)
0929 11:28:02.242882 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:02.242887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:02.251933 (+ 9046us) server_negotiation.cc:407] Connection header received
0929 11:28:02.252142 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:02.252148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:02.252276 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:02.252395 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:28:02.254629 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:02.255858 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:02.258596 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:02.259600 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:02.259786 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:02.268160 (+ 8374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:28:02.268198 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:28:02.268221 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:28:02.268296 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:28:02.272644 (+ 4348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:02.273664 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:02.273682 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:02.273686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:02.273891 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:02.274384 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:02.274389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:02.274392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:02.274904 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:28:02.275287 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:02.275854 (+ 567us) server_negotiation.cc:299] Negotiation successful
0929 11:28:02.276253 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:28:02.277633 2262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:28:02.383608 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 2139
I20260929 11:28:02.401925 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 2273
Sep 29 11:28:02 dist-test-slave-qk05 krb5kdc[2033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.2@KRBTEST.COM
I20260929 11:28:02.418776 2552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:02.402914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56078 (local address 127.23.153.2:45079)
0929 11:28:02.403123 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:02.403127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:02.403146 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:28:02.403213 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:02.403217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:02.403291 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:02.403381 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:28:02.404627 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:02.405414 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:02.406785 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:02.407325 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:02.407438 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:02.411867 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:28:02.411884 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:28:02.411902 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:28:02.411952 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:28:02.414994 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:02.416114 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:02.416127 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:02.416130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:02.416424 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:02.417003 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:02.417009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:02.417012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:02.417671 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:28:02.417950 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:02.418283 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 11:28:02.418491 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 11:28:02.419914 2396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:28:02.513681 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 2273
I20260929 11:28:02.528808 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 2407
Sep 29 11:28:02 dist-test-slave-qk05 krb5kdc[2033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.3@KRBTEST.COM
I20260929 11:28:02.545258 2549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:02.529582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.23.153.3:36643)
0929 11:28:02.529748 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:02.529753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:02.529773 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:28:02.529858 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:02.529861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:02.529924 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:02.530023 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:28:02.531440 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:02.532219 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:02.533574 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:02.533987 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:02.534096 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:02.539248 (+ 5152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:28:02.539266 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:28:02.539283 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:28:02.539330 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:28:02.542073 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:02.542887 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:02.542896 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:02.542898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:02.543168 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:02.543847 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:02.543851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:02.543853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:02.544235 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:28:02.544472 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:02.544819 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 11:28:02.544997 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 11:28:02.545925 2530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:28:02.635742 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 2407
I20260929 11:28:02.651335 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 2046
I20260929 11:28:02.652467 2108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:28:02.766280 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 2046
2026-09-29T11:28:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4497 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:28:02 dist-test-slave-qk05 krb5kdc[2589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:28:02 dist-test-slave-qk05 krb5kdc[2589](info): set up 2 sockets
Sep 29 11:28:02 dist-test-slave-qk05 krb5kdc[2589](info): commencing operation
krb5kdc: starting...
W20260929 11:28:03.819736 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:28:03 dist-test-slave-qk05 krb5kdc[2589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:28:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:28:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.153.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.153.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.153.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/rangeradmin_127.23.153.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.153.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/rangeradmin_127.23.153.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.153.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.153.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.153.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/rangerlookup_127.23.153.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.153.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/rangerlookup_127.23.153.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.153.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.153.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.153.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/HTTP_127.23.153.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.153.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/HTTP_127.23.153.20@KRBTEST.COM.keytab.
I20260929 11:28:04.182209 24164 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBSjPgi/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/postgres -l logfile start
2026-09-29 11:28:06.878 UTC [2619] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 11:28:06.878 UTC [2619] LOG: listening on IPv4 address "127.23.153.20", port 59149
2026-09-29 11:28:06.882 UTC [2619] LOG: listening on Unix socket "/tmp/.s.PGSQL.59149"
2026-09-29 11:28:06.904 UTC [2622] LOG: database system was shut down at 2026-09-29 11:28:05 UTC
2026-09-29 11:28:06.908 UTC [2619] LOG: database system is ready to accept connections
I20260929 11:28:07.827425 24164 mini_postgres.cc:96] Postgres bound to 59149
2026-09-29 11:28:07.845 UTC [2630] FATAL: database "slave" does not exist
127.23.153.20:59149 - accepting connections
I20260929 11:28:07.846385 24164 mini_ranger.cc:161] Starting Ranger...
I20260929 11:28:07.872931 24164 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 11:28:08.389429 24164 mini_ranger.cc:87] Created ranger Postgres database
I20260929 11:28:08.389621 24164 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ranger-admin
2026-09-29 11:28:08,710 [I] DB FLAVOR :POSTGRES
2026-09-29 11:28:08,711 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 11:28:08,711 [I] Checking connection..
2026-09-29 11:28:08,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 11:28:09,002 [I] Checking connection passed.
2026-09-29 11:28:09,003 [I] --------- Verifying version history table ---------
2026-09-29 11:28:09,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 11:28:09,282 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 11:28:09,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 11:28:09,543 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 11:28:09,543 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 11:28:09,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 11:28:09.812 UTC [2694] WARNING: there is no transaction in progress
2026-09-29 11:28:09,818 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 11:28:09,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 11:28:10,091 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 11:28:10,091 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 11:28:10,091 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 11:28:10,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 11:28:10,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qk05.c.gcp-upstream.internal','N') ;"
2026-09-29 11:28:10,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 11:28:11,207 [I] Table x_portal_user does not exist in database ranger
2026-09-29 11:28:11,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 11:28:11,483 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 11:28:11,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 11:28:11,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 11:28:12,019 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 11:28:12,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 11:28:13.100 UTC [2802] WARNING: there is no transaction in progress
2026-09-29 11:28:13.118 UTC [2802] WARNING: there is no transaction in progress
2026-09-29 11:28:13.138 UTC [2802] WARNING: there is no transaction in progress
2026-09-29 11:28:13.154 UTC [2802] WARNING: there is no transaction in progress
2026-09-29 11:28:13.173 UTC [2802] WARNING: there is no transaction in progress
2026-09-29 11:28:13.193 UTC [2802] WARNING: there is no transaction in progress
2026-09-29 11:28:13.257 UTC [2802] WARNING: there is no transaction in progress
2026-09-29 11:28:13.265 UTC [2802] WARNING: there is no transaction in progress
2026-09-29 11:28:13.283 UTC [2802] WARNING: there is no transaction in progress
2026-09-29 11:28:13.296 UTC [2802] WARNING: there is no transaction in progress
2026-09-29 11:28:13.310 UTC [2802] WARNING: there is no transaction in progress
2026-09-29 11:28:13.326 UTC [2802] WARNING: there is no transaction in progress
2026-09-29 11:28:13.337 UTC [2802] WARNING: there is no transaction in progress
2026-09-29 11:28:13.346 UTC [2802] WARNING: there is no transaction in progress
2026-09-29 11:28:13.354 UTC [2802] WARNING: there is no transaction in progress
2026-09-29 11:28:13.363 UTC [2802] WARNING: there is no transaction in progress
2026-09-29 11:28:14,137 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 11:28:14,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 11:28:14,387 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 11:28:14,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 11:28:14,649 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 11:28:14,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 11:28:14,921 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 11:28:14,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 11:28:15,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 11:28:15,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qk05.c.gcp-upstream.internal';"
2026-09-29 11:28:15,710 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 11:28:15,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 11:28:15,981 [I] DB_PATCHES have already been applied
I20260929 11:28:15.987449 24164 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBSjPgi/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 11:28:15.987546 24164 mini_ranger.cc:193] Using host: 127.23.153.20
I20260929 11:28:16.002637 24164 mini_ranger.cc:227] Ranger admin URL: http://127.23.153.20:53509
Sep 29, 2026 11:28:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 11:28:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 11:28:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 11:28:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 11:28:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 11:28:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 11:28:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 11:28:17.086898 24164 mini_ranger.cc:160] Time spent starting Ranger: real 9.241s user 0.008s sys 0.037s
Sep 29, 2026 11:28:40 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 11:28:40 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 11:28:40 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 11:28:40 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 11:28:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 11:28:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 11:28:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 11:28:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 11:28:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 11:28:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 11:28:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 11:28:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 11:28:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 11:28:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 11:28:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 11:28:40 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 11:28:40 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 11:28:41 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 11:28:42.437884 24164 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.153.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.153.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.153.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/rangerkms_127.23.153.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.153.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/rangerkms_127.23.153.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.153.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.153.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.153.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/HTTP_127.23.153.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.153.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/HTTP_127.23.153.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 11:28:42 dist-test-slave-qk05 krb5kdc[2589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681322, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 11:28:42.609494 24164 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 11:28:42.636620 24164 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 11:28:43.607709 24164 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 11:28:43.607892 24164 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ranger-kms
I20260929 11:28:43.608014 24164 mini_ranger_kms.cc:227] Using postgres at 127.23.153.20:59149
2026-09-29 11:28:44,081 [I] DB FLAVOR :POSTGRES
2026-09-29 11:28:44,082 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 11:28:44,082 [I] Checking connection
2026-09-29 11:28:44,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 11:28:44,384 [I] connection success
2026-09-29 11:28:44,384 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 11:28:44,384 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 11:28:44,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 11:28:44,714 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 11:28:44,714 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 11:28:44,714 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 11:28:44,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:59149/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 11:28:45,100 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 11:28:45.744446 24164 mini_ranger_kms.cc:310] Created kms service
I20260929 11:28:46.026069 24164 mini_ranger_kms.cc:326] Created kudu user
I20260929 11:28:46.083163 24164 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 11:28:46 dist-test-slave-qk05 krb5kdc[2589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681322, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
I20260929 11:28:46.573884 24164 mini_ranger_kms.cc:384] Added ranger policy
I20260929 11:28:46.574175 24164 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 11:28:46.574218 24164 mini_ranger_kms.cc:240] Using host: 127.23.153.20
I20260929 11:28:46.599812 24164 mini_ranger_kms.cc:276] Ranger KMS PID: 2983
I20260929 11:28:46.600009 24164 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.153.20:38493
Sep 29, 2026 11:28:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 11:28:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 11:28:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 11:28:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 11:28:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 11:28:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 11:28:48.662777 24164 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.053s user 0.142s sys 0.352s
I20260929 11:28:48.663087 24164 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 11:28:48.663331 24164 mini_ranger_kms.cc:401] 127.23.153.20:38493/kms/v1/keys
Sep 29 11:28:48 dist-test-slave-qk05 krb5kdc[2589](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790681328, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.153.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 11:28:49 dist-test-slave-qk05 krb5kdc[2589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681328, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.153.20@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
Sep 29 11:28:49 dist-test-slave-qk05 krb5kdc[2589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681328, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.153.20@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
Sep 29 11:28:52 dist-test-slave-qk05 krb5kdc[2589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:28:52.450580 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:35795
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:35795
--ranger_config_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.153.20:38493/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:28:52.824926 3033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:28:52.825250 3033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:28:52.825363 3033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:28:52.833429 3033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:28:52.833583 3033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:28:52.833630 3033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:28:52.833703 3033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:28:52.833767 3033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:28:52.844907 3033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.153.20:38493/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:35795
--ranger_config_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:35795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:28:52.846278 3033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:28:52.847765 3033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:28:52.862061 3041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:28:52.862169 3038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:28:52.862316 3039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:28:52.863197 3033 server_base.cc:1034] running on GCE node
I20260929 11:28:52.864203 3033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:28:52.866228 3033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:28:52.867496 3033 hybrid_clock.cc:648] HybridClock initialized: now 1790681332867430 us; error 51 us; skew 500 ppm
I20260929 11:28:52.867944 3033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:28:52 dist-test-slave-qk05 krb5kdc[2589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681332, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:28:52.875414 3033 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:28:52.877390 3033 webserver.cc:460] Webserver started at http://127.23.153.62:33237/ using document root <none> and password file <none>
I20260929 11:28:52.878314 3033 fs_manager.cc:359] Metadata directory not provided
I20260929 11:28:52.878465 3033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:28:52.878844 3033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:28:52 dist-test-slave-qk05 krb5kdc[2589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681332, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
I20260929 11:28:52.943686 3033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "43ad644a4cac4e9392cf8b324aac83b2"
format_stamp: "Formatted at 2026-09-29 11:28:52 on dist-test-slave-qk05"
server_key: "89c309cad6b31ed248b6e3e30f213a51"
server_key_iv: "f654f87671cea9276ae16e6d09f4eb67"
server_key_version: "kuduclusterkey@0"
I20260929 11:28:52.945087 3033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "43ad644a4cac4e9392cf8b324aac83b2"
format_stamp: "Formatted at 2026-09-29 11:28:52 on dist-test-slave-qk05"
server_key: "89c309cad6b31ed248b6e3e30f213a51"
server_key_iv: "f654f87671cea9276ae16e6d09f4eb67"
server_key_version: "kuduclusterkey@0"
I20260929 11:28:52.955410 3033 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.010s sys 0.000s
I20260929 11:28:52.980371 3048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:28:52.982430 3033 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260929 11:28:52.982681 3033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "43ad644a4cac4e9392cf8b324aac83b2"
format_stamp: "Formatted at 2026-09-29 11:28:52 on dist-test-slave-qk05"
server_key: "89c309cad6b31ed248b6e3e30f213a51"
server_key_iv: "f654f87671cea9276ae16e6d09f4eb67"
server_key_version: "kuduclusterkey@0"
I20260929 11:28:52.982981 3033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:28:53.003129 3033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:28:53.008679 3033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:28:53.009068 3033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:28:53.030853 3099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:35795 every 8 connection(s)
I20260929 11:28:53.030874 3033 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:35795
I20260929 11:28:53.033025 3033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:28:53.037784 3100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:28:53.041213 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 3033
I20260929 11:28:53.041440 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:28:53.042351 24164 external_mini_cluster.cc:1428] Setting key a3e923e0fc9934f8629cc9c9250b107b
I20260929 11:28:53.054153 3100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2: Bootstrap starting.
I20260929 11:28:53.060052 3100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2: Neither blocks nor log segments found. Creating new log.
I20260929 11:28:53.061261 3100 log.cc:824] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2: Log is configured to *not* fsync() on all Append() calls
I20260929 11:28:53.064680 3100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2: No bootstrap required, opened a new log
Sep 29 11:28:53 dist-test-slave-qk05 krb5kdc[2589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:28:53.074401 3100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ad644a4cac4e9392cf8b324aac83b2" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 35795 } }
I20260929 11:28:53.074935 3100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:28:53.075088 3100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ad644a4cac4e9392cf8b324aac83b2, State: Initialized, Role: FOLLOWER
I20260929 11:28:53.075855 3100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ad644a4cac4e9392cf8b324aac83b2" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 35795 } }
I20260929 11:28:53.076116 3100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:28:53.076263 3100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:28:53.076442 3100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:28:53.078990 3100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ad644a4cac4e9392cf8b324aac83b2" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 35795 } }
I20260929 11:28:53.079690 3100 leader_election.cc:302] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43ad644a4cac4e9392cf8b324aac83b2; no voters:
I20260929 11:28:53.080459 3100 leader_election.cc:288] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:28:53.080638 3105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:28:53.081835 3105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2 [term 1 LEADER]: Becoming Leader. State: Replica: 43ad644a4cac4e9392cf8b324aac83b2, State: Running, Role: LEADER
I20260929 11:28:53.082505 3105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ad644a4cac4e9392cf8b324aac83b2" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 35795 } }
I20260929 11:28:53.083171 3100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:28:53.084872 3103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:53.044925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.23.153.62:35795)
0929 11:28:53.059787 (+ 14862us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:53.059806 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:53.059864 (+ 58us) server_negotiation.cc:407] Connection header received
0929 11:28:53.062595 (+ 2731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:53.062633 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:53.063285 (+ 652us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:53.063824 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:28:53.065450 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:53.067803 (+ 2353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:53.069787 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:53.070655 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:53.070852 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:53.075842 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:28:53.075852 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:28:53.075869 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:28:53.075916 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:28:53.079174 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:53.080098 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:53.080107 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:53.080114 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:53.080354 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:53.080987 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:53.080993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:53.080996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:53.081921 (+ 925us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:28:53.082226 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:53.082797 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 11:28:53.083283 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14575,"thread_start_us":14307,"threads_started":1}
I20260929 11:28:53.085707 3107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43ad644a4cac4e9392cf8b324aac83b2. Latest consensus state: current_term: 1 leader_uuid: "43ad644a4cac4e9392cf8b324aac83b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ad644a4cac4e9392cf8b324aac83b2" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 35795 } } }
I20260929 11:28:53.086198 3107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2 [sys.catalog]: This master's current role is: LEADER
I20260929 11:28:53.086720 3106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43ad644a4cac4e9392cf8b324aac83b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43ad644a4cac4e9392cf8b324aac83b2" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 35795 } } }
I20260929 11:28:53.087049 3106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2 [sys.catalog]: This master's current role is: LEADER
I20260929 11:28:53.089794 3111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:28:53.095813 3111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:28:53.096940 3100 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 11:28:53.115613 3111 catalog_manager.cc:1347] Generated new cluster ID: 36033c93e1274cf3a020c638fd65ff58
I20260929 11:28:53.115777 3111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:28:53.145179 3111 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:28:53.147454 3111 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:28:53.159904 3111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2: Generated new TSK 0
I20260929 11:28:53.161547 3111 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
I20260929 11:28:53.962894 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.153.20:38493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:35795
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:28:54 dist-test-slave-qk05 krb5kdc[2589](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790681334, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 11:28:54.428311 3145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:28:54.428858 3145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:28:54.429033 3145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:28:54.441391 3145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:28:54.441572 3145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:28:54.441876 3145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:28:54.458755 3145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.153.20:38493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:35795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:28:54.460789 3145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:28:54.462697 3145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:28:54.490020 3156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:28:54.490958 3155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:28:54.504652 3158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:28:54.505149 3145 server_base.cc:1034] running on GCE node
I20260929 11:28:54.505980 3145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:28:54.511449 3145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:28:54.512877 3145 hybrid_clock.cc:648] HybridClock initialized: now 1790681334512743 us; error 136 us; skew 500 ppm
I20260929 11:28:54.513348 3145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:28:54 dist-test-slave-qk05 krb5kdc[2589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681334, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:28:54.518657 3145 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:28:54.521224 3145 webserver.cc:460] Webserver started at http://127.23.153.1:34075/ using document root <none> and password file <none>
I20260929 11:28:54.522408 3145 fs_manager.cc:359] Metadata directory not provided
I20260929 11:28:54.522606 3145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:28:54.523087 3145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:28:54 dist-test-slave-qk05 krb5kdc[2589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681334, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
I20260929 11:28:54.546922 3145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "a6076fee79e342bc89a4e251883d3b3c"
format_stamp: "Formatted at 2026-09-29 11:28:54 on dist-test-slave-qk05"
server_key: "36648fb57cb589393a94ab4ee341343a"
server_key_iv: "ed11492bfaf9c2336c9d7cba251156df"
server_key_version: "kuduclusterkey@0"
I20260929 11:28:54.550868 3145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "a6076fee79e342bc89a4e251883d3b3c"
format_stamp: "Formatted at 2026-09-29 11:28:54 on dist-test-slave-qk05"
server_key: "36648fb57cb589393a94ab4ee341343a"
server_key_iv: "ed11492bfaf9c2336c9d7cba251156df"
server_key_version: "kuduclusterkey@0"
I20260929 11:28:54.563479 3145 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.005s sys 0.004s
I20260929 11:28:54.583148 3165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:28:54.585426 3145 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260929 11:28:54.585668 3145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "a6076fee79e342bc89a4e251883d3b3c"
format_stamp: "Formatted at 2026-09-29 11:28:54 on dist-test-slave-qk05"
server_key: "36648fb57cb589393a94ab4ee341343a"
server_key_iv: "ed11492bfaf9c2336c9d7cba251156df"
server_key_version: "kuduclusterkey@0"
I20260929 11:28:54.585968 3145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:28:54.616576 3145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:28:54.623739 3145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:28:54.624181 3145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:28:54.625538 3145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:28:54.627153 3145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:28:54.627242 3145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:28:54.627395 3145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:28:54.627449 3145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:28:54.695395 3145 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:35561
I20260929 11:28:54.695407 3277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:35561 every 8 connection(s)
I20260929 11:28:54.697202 3145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:28:54.699962 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 3145
I20260929 11:28:54.700194 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:28:54.701053 24164 external_mini_cluster.cc:1428] Setting key 1c4ea59f569fa31310be8164c96b1e10
Sep 29 11:28:54 dist-test-slave-qk05 krb5kdc[2589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681334, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.153.2@KRBTEST.COM; defaulting to no policy
I20260929 11:28:54.764772 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:54.701938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:45105 (local address 127.23.153.62:35795)
0929 11:28:54.725275 (+ 23337us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:54.725284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:54.729093 (+ 3809us) server_negotiation.cc:407] Connection header received
0929 11:28:54.729200 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:54.729206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:54.729323 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:54.729435 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:28:54.747700 (+ 18265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:54.748847 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:54.750851 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:54.751567 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:54.751718 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:54.756588 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:28:54.756597 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:28:54.756602 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:28:54.756649 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:28:54.760318 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:54.761508 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:54.761515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:54.761519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:54.761720 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:54.762391 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:54.762397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:54.762401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:54.762803 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:28:54.762953 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:54.764106 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0929 11:28:54.764381 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23148,"thread_start_us":27616,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.2@KRBTEST.COM" created.
I20260929 11:28:54.767290 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:54.702349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:35795 (local address 127.23.153.1:45105)
0929 11:28:54.724644 (+ 22295us) negotiation.cc:107] Waiting for socket to connect
0929 11:28:54.725388 (+ 744us) client_negotiation.cc:174] Beginning negotiation
0929 11:28:54.728525 (+ 3137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:28:54.745357 (+ 16832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:28:54.745373 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:28:54.746367 (+ 994us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:28:54.747364 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:54.747406 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:54.749093 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:54.749097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:54.750637 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:54.750655 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:54.752105 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:54.752113 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:54.752452 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:54.753299 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:28:54.753332 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:28:54.756235 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:28:54.760591 (+ 4356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:28:54.760604 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:28:54.760628 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:28:54.761333 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:28:54.761915 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:28:54.761920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:28:54.761925 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:28:54.762250 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:28:54.763100 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:28:54.763109 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:28:54.763610 (+ 501us) client_negotiation.cc:769] Sending connection context
0929 11:28:54.765101 (+ 1491us) client_negotiation.cc:240] Negotiation successful
0929 11:28:54.765767 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21903,"thread_start_us":20508,"threads_started":1}
I20260929 11:28:54.769445 3278 heartbeater.cc:347] Connected to a master server at 127.23.153.62:35795
I20260929 11:28:54.769815 3278 heartbeater.cc:464] Registering TS with master...
I20260929 11:28:54.770710 3278 heartbeater.cc:511] Master 127.23.153.62:35795 requested a full tablet report, sending...
I20260929 11:28:54.773514 3065 ts_manager.cc:194] Registered new tserver with Master: a6076fee79e342bc89a4e251883d3b3c (127.23.153.1:35561)
I20260929 11:28:54.775645 3065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:45105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
I20260929 11:28:54.808996 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.153.20:38493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:35795
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:28:54 dist-test-slave-qk05 krb5kdc[2589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681334, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
Sep 29 11:28:55 dist-test-slave-qk05 krb5kdc[2589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681334, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
W20260929 11:28:55.312733 3284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:28:55.313277 3284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:28:55.313493 3284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:28:55.325855 3284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:28:55.326123 3284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:28:55.326378 3284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:28:55.345027 3284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.153.20:38493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:35795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:28:55.347219 3284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:28:55.349732 3284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:28:55.393117 3291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:28:55.402798 3292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:28:55.403643 3295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:28:55.404238 3284 server_base.cc:1034] running on GCE node
I20260929 11:28:55.405333 3284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:28:55.407315 3284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:28:55.412307 3284 hybrid_clock.cc:648] HybridClock initialized: now 1790681335412155 us; error 105 us; skew 500 ppm
I20260929 11:28:55.413205 3284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:28:55 dist-test-slave-qk05 krb5kdc[2589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681335, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:28:55.419776 3284 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:28:55.429592 3284 webserver.cc:460] Webserver started at http://127.23.153.2:36919/ using document root <none> and password file <none>
I20260929 11:28:55.431870 3284 fs_manager.cc:359] Metadata directory not provided
I20260929 11:28:55.432235 3284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:28:55.432804 3284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:28:55 dist-test-slave-qk05 krb5kdc[2589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681335, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
I20260929 11:28:55.468607 3284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "4b6bafa329464194b3c265b4afc386b0"
format_stamp: "Formatted at 2026-09-29 11:28:55 on dist-test-slave-qk05"
server_key: "118b52b8aee9519a8a0c42c2ad1f2f22"
server_key_iv: "35f8b4b6927e38a287024fbb2af4c248"
server_key_version: "kuduclusterkey@0"
I20260929 11:28:55.469933 3284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "4b6bafa329464194b3c265b4afc386b0"
format_stamp: "Formatted at 2026-09-29 11:28:55 on dist-test-slave-qk05"
server_key: "118b52b8aee9519a8a0c42c2ad1f2f22"
server_key_iv: "35f8b4b6927e38a287024fbb2af4c248"
server_key_version: "kuduclusterkey@0"
I20260929 11:28:55.493304 3284 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.010s sys 0.011s
I20260929 11:28:55.514662 3301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:28:55.517692 3284 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260929 11:28:55.517977 3284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "4b6bafa329464194b3c265b4afc386b0"
format_stamp: "Formatted at 2026-09-29 11:28:55 on dist-test-slave-qk05"
server_key: "118b52b8aee9519a8a0c42c2ad1f2f22"
server_key_iv: "35f8b4b6927e38a287024fbb2af4c248"
server_key_version: "kuduclusterkey@0"
I20260929 11:28:55.518281 3284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:28:55.568202 3284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:28:55.574079 3284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:28:55.574448 3284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:28:55.575755 3284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:28:55.577656 3284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:28:55.577762 3284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:28:55.577888 3284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:28:55.577946 3284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:28:55.617597 3420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:36531 every 8 connection(s)
I20260929 11:28:55.617740 3284 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:36531
I20260929 11:28:55.619711 3284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:28:55.624531 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 3284
I20260929 11:28:55.624708 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:28:55.625269 24164 external_mini_cluster.cc:1428] Setting key 3ba1789284c37bb0a02668e887350508
Sep 29 11:28:55 dist-test-slave-qk05 krb5kdc[2589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681335, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:28:55.644526 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:55.622745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:35379 (local address 127.23.153.62:35795)
0929 11:28:55.623273 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:55.623279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:55.624592 (+ 1313us) server_negotiation.cc:407] Connection header received
0929 11:28:55.626731 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:55.626737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:55.626855 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:55.626977 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:28:55.629150 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:55.630253 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:55.632745 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:55.633575 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:55.633751 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:55.638274 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:28:55.638279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:28:55.638282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:28:55.638323 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:28:55.640975 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:55.641855 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:55.641860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:55.641862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:55.642010 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:55.642508 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:55.642512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:55.642514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:55.642873 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:28:55.643043 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:55.643814 (+ 771us) server_negotiation.cc:299] Negotiation successful
0929 11:28:55.644084 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":264,"threads_started":1}
I20260929 11:28:55.645596 3424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:55.623380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:35795 (local address 127.23.153.2:35379)
0929 11:28:55.624241 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0929 11:28:55.624303 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 11:28:55.626401 (+ 2098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:28:55.627314 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:28:55.627339 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:28:55.628012 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:28:55.628905 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:55.628936 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:55.630463 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:55.630468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:55.632492 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:55.632514 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:55.633885 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:55.633893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:55.634215 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:55.635471 (+ 1256us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:28:55.635506 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:28:55.637955 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:28:55.641172 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:28:55.641182 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:28:55.641200 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:28:55.641694 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:28:55.642132 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:28:55.642137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:28:55.642140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:28:55.642380 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:28:55.643006 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:28:55.643039 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:28:55.643422 (+ 383us) client_negotiation.cc:769] Sending connection context
0929 11:28:55.643688 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 11:28:55.644255 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":317,"threads_started":1}
I20260929 11:28:55.647181 3421 heartbeater.cc:347] Connected to a master server at 127.23.153.62:35795
I20260929 11:28:55.647511 3421 heartbeater.cc:464] Registering TS with master...
I20260929 11:28:55.648316 3421 heartbeater.cc:511] Master 127.23.153.62:35795 requested a full tablet report, sending...
I20260929 11:28:55.649894 3065 ts_manager.cc:194] Registered new tserver with Master: 4b6bafa329464194b3c265b4afc386b0 (127.23.153.2:36531)
I20260929 11:28:55.650723 3065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:35379
WARNING: no policy specified for kudu/127.23.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
I20260929 11:28:55.686951 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.153.20:38493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:35795
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:28:55.780040 3278 heartbeater.cc:503] Master 127.23.153.62:35795 was elected leader, sending a full tablet report...
W20260929 11:28:56.065061 3427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:28:56.065395 3427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:28:56.065492 3427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:28:56.075249 3427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:28:56.075397 3427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:28:56.075520 3427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:28:56.087296 3427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.153.20:38493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:35795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:28:56.088970 3427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:28:56.090869 3427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:28:56.107198 3435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:28:56.107215 3432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:28:56.107257 3433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:28:56.113250 3427 server_base.cc:1034] running on GCE node
I20260929 11:28:56.114144 3427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:28:56.115283 3427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:28:56.116524 3427 hybrid_clock.cc:648] HybridClock initialized: now 1790681336116494 us; error 95 us; skew 500 ppm
I20260929 11:28:56.116869 3427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:28:56 dist-test-slave-qk05 krb5kdc[2589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681336, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:28:56.120223 3427 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:28:56.121971 3427 webserver.cc:460] Webserver started at http://127.23.153.3:45411/ using document root <none> and password file <none>
I20260929 11:28:56.122778 3427 fs_manager.cc:359] Metadata directory not provided
I20260929 11:28:56.122918 3427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:28:56.123323 3427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:28:56 dist-test-slave-qk05 krb5kdc[2589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681336, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
I20260929 11:28:56.140858 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "6c6979c4e0374f9fb49345e32bc6ab29"
format_stamp: "Formatted at 2026-09-29 11:28:56 on dist-test-slave-qk05"
server_key: "6c926ca80b01ba32ab5b989960890d60"
server_key_iv: "ef06a410cc33881a7b2c5caaef2f9b77"
server_key_version: "kuduclusterkey@0"
I20260929 11:28:56.142345 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "6c6979c4e0374f9fb49345e32bc6ab29"
format_stamp: "Formatted at 2026-09-29 11:28:56 on dist-test-slave-qk05"
server_key: "6c926ca80b01ba32ab5b989960890d60"
server_key_iv: "ef06a410cc33881a7b2c5caaef2f9b77"
server_key_version: "kuduclusterkey@0"
I20260929 11:28:56.149694 3427 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260929 11:28:56.168723 3442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:28:56.170392 3427 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:28:56.170588 3427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "6c6979c4e0374f9fb49345e32bc6ab29"
format_stamp: "Formatted at 2026-09-29 11:28:56 on dist-test-slave-qk05"
server_key: "6c926ca80b01ba32ab5b989960890d60"
server_key_iv: "ef06a410cc33881a7b2c5caaef2f9b77"
server_key_version: "kuduclusterkey@0"
I20260929 11:28:56.170816 3427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:28:56.199831 3427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:28:56.204921 3427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:28:56.205263 3427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:28:56.206305 3427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:28:56.207849 3427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:28:56.207935 3427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:28:56.208031 3427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:28:56.208076 3427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:28:56.242456 3427 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:42213
I20260929 11:28:56.242671 3554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:42213 every 8 connection(s)
I20260929 11:28:56.243943 3427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:28:56.245935 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 3427
I20260929 11:28:56.246119 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:28:56.246673 24164 external_mini_cluster.cc:1428] Setting key 46b84682212b90188171b2b34aa3274a
Sep 29 11:28:56 dist-test-slave-qk05 krb5kdc[2589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681336, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:28:56.265151 3557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:56.246778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:40651 (local address 127.23.153.62:35795)
0929 11:28:56.247289 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:56.247294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:56.248994 (+ 1700us) server_negotiation.cc:407] Connection header received
0929 11:28:56.250883 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:56.250890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:56.250971 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:56.251108 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:28:56.252594 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.253334 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.254995 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.255481 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.255591 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:56.258952 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:28:56.258958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:28:56.258962 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:28:56.258999 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:28:56.261948 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:56.262958 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:56.262963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:56.262965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:56.263153 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:56.263554 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:56.263558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:56.263560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:56.263849 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:28:56.263970 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:56.264674 (+ 704us) server_negotiation.cc:299] Negotiation successful
0929 11:28:56.264856 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":266,"threads_started":1}
I20260929 11:28:56.266084 3558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:56.247562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:35795 (local address 127.23.153.3:40651)
0929 11:28:56.248833 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0929 11:28:56.248881 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 11:28:56.250599 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:28:56.251313 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:28:56.251322 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:28:56.251776 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:28:56.252400 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:56.252420 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.253499 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.253502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:56.254835 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:56.254848 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.255678 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.255683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:56.255890 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:56.256587 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:28:56.256612 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:28:56.258680 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:28:56.262125 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:28:56.262133 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:28:56.262149 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:28:56.262768 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:28:56.263249 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:28:56.263253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:28:56.263255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:28:56.263452 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:28:56.263949 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:28:56.263955 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:28:56.264310 (+ 355us) client_negotiation.cc:769] Sending connection context
0929 11:28:56.264566 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 11:28:56.264974 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"thread_start_us":352,"threads_started":1}
I20260929 11:28:56.267633 3555 heartbeater.cc:347] Connected to a master server at 127.23.153.62:35795
I20260929 11:28:56.267951 3555 heartbeater.cc:464] Registering TS with master...
I20260929 11:28:56.268714 3555 heartbeater.cc:511] Master 127.23.153.62:35795 requested a full tablet report, sending...
I20260929 11:28:56.270368 3065 ts_manager.cc:194] Registered new tserver with Master: 6c6979c4e0374f9fb49345e32bc6ab29 (127.23.153.3:42213)
I20260929 11:28:56.271256 3065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:40651
I20260929 11:28:56.275424 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:28:56.295578 3557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:56.281434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36744 (local address 127.23.153.62:35795)
0929 11:28:56.281664 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:56.281671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:56.282182 (+ 511us) server_negotiation.cc:407] Connection header received
0929 11:28:56.282378 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:56.282382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:56.282455 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:56.282550 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:28:56.284155 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.284941 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.286256 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.286699 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.286814 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:56.289375 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:28:56.289382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:28:56.289386 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:28:56.289431 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:28:56.292525 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:56.293388 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:56.293393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:56.293395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:56.293543 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:56.293925 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:56.293930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:56.293932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:56.294318 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:28:56.294529 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:56.294978 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 11:28:56.295250 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 11:28:56.301297 3065 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:28:56.309120 3065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:28:56.336865 3568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:56.325249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47118 (local address 127.23.153.2:36531)
0929 11:28:56.325846 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:56.325855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:56.326673 (+ 818us) server_negotiation.cc:407] Connection header received
0929 11:28:56.327001 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:56.327007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:56.327260 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:56.327471 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:28:56.328403 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.330289 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.333570 (+ 3281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.335159 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.335302 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:56.335525 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:56.336327 (+ 802us) server_negotiation.cc:299] Negotiation successful
0929 11:28:56.336524 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":308,"threads_started":1}
I20260929 11:28:56.336934 3569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:56.325196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:36531 (local address 127.0.0.1:47118)
0929 11:28:56.326520 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0929 11:28:56.326563 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 11:28:56.326763 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:28:56.327793 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:28:56.327805 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:28:56.327836 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:28:56.328243 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:56.328253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.330481 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.330487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:56.333318 (+ 2831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:56.333344 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.335686 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.335692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:56.335935 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:56.336000 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 11:28:56.336131 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 11:28:56.336397 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"mutex_wait_us":26,"thread_start_us":1166,"threads_started":1}
I20260929 11:28:56.340219 3572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:56.326161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:42213 (local address 127.0.0.1:48938)
0929 11:28:56.327923 (+ 1762us) negotiation.cc:107] Waiting for socket to connect
0929 11:28:56.327946 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:28:56.328070 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:28:56.332109 (+ 4039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:28:56.332117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:28:56.332158 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:28:56.332610 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:56.332621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.334993 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.334999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:56.336854 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:56.336870 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.339135 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.339143 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:56.339330 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:56.339364 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 11:28:56.339455 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 11:28:56.339806 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1634,"mutex_wait_us":27,"thread_start_us":4070,"threads_started":1}
I20260929 11:28:56.340524 3573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:56.327878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38286 (local address 127.23.153.1:35561)
0929 11:28:56.330074 (+ 2196us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:56.330084 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:56.330707 (+ 623us) server_negotiation.cc:407] Connection header received
0929 11:28:56.330886 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:56.330896 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:56.331238 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:56.331457 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:28:56.332451 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.334044 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.336860 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.338494 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.338650 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:56.338902 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:56.339786 (+ 884us) server_negotiation.cc:299] Negotiation successful
0929 11:28:56.339986 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1996,"thread_start_us":1860,"threads_started":1}
I20260929 11:28:56.341184 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:56.326168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48938 (local address 127.23.153.3:42213)
0929 11:28:56.331305 (+ 5137us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:56.331314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:56.331345 (+ 31us) server_negotiation.cc:407] Connection header received
0929 11:28:56.331434 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:56.331442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:56.331846 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:56.332074 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:28:56.332852 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.334632 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.337081 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.338495 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.340424 (+ 1929us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:56.340708 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:56.340791 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 11:28:56.340923 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4964,"thread_start_us":4785,"threads_started":1}
I20260929 11:28:56.341284 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:56.327370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:35561 (local address 127.0.0.1:38286)
0929 11:28:56.330561 (+ 3191us) negotiation.cc:107] Waiting for socket to connect
0929 11:28:56.330580 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:28:56.330721 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:28:56.331466 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:28:56.331471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:28:56.331605 (+ 134us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:28:56.332159 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:56.332174 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.334220 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.334226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:56.336613 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:56.336631 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.339160 (+ 2529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.339168 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:56.339412 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:56.339447 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:28:56.339644 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 11:28:56.340912 (+ 1268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3026,"mutex_wait_us":1190,"thread_start_us":996,"threads_started":1}
I20260929 11:28:56.342163 3356 tablet_service.cc:1467] Processing CreateTablet for tablet cb7fd58e99764ea38d6d95630776ff9b (DEFAULT_TABLE table=test-table [id=c47f2c50b2694ec6894303bacafeec06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:28:56.343894 3356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7fd58e99764ea38d6d95630776ff9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:28:56.345001 3213 tablet_service.cc:1467] Processing CreateTablet for tablet cb7fd58e99764ea38d6d95630776ff9b (DEFAULT_TABLE table=test-table [id=c47f2c50b2694ec6894303bacafeec06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:28:56.344985 3490 tablet_service.cc:1467] Processing CreateTablet for tablet cb7fd58e99764ea38d6d95630776ff9b (DEFAULT_TABLE table=test-table [id=c47f2c50b2694ec6894303bacafeec06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:28:56.346428 3213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7fd58e99764ea38d6d95630776ff9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:28:56.346491 3490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7fd58e99764ea38d6d95630776ff9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:28:56.360659 3574 tablet_bootstrap.cc:492] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c: Bootstrap starting.
I20260929 11:28:56.363044 3576 tablet_bootstrap.cc:492] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29: Bootstrap starting.
I20260929 11:28:56.364837 3574 tablet_bootstrap.cc:654] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c: Neither blocks nor log segments found. Creating new log.
I20260929 11:28:56.366214 3575 tablet_bootstrap.cc:492] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0: Bootstrap starting.
I20260929 11:28:56.366418 3574 log.cc:824] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c: Log is configured to *not* fsync() on all Append() calls
I20260929 11:28:56.367156 3576 tablet_bootstrap.cc:654] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29: Neither blocks nor log segments found. Creating new log.
I20260929 11:28:56.368976 3576 log.cc:824] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29: Log is configured to *not* fsync() on all Append() calls
I20260929 11:28:56.370431 3575 tablet_bootstrap.cc:654] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0: Neither blocks nor log segments found. Creating new log.
I20260929 11:28:56.371291 3574 tablet_bootstrap.cc:492] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c: No bootstrap required, opened a new log
I20260929 11:28:56.371624 3574 ts_tablet_manager.cc:1397] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260929 11:28:56.371897 3575 log.cc:824] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0: Log is configured to *not* fsync() on all Append() calls
I20260929 11:28:56.372859 3576 tablet_bootstrap.cc:492] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29: No bootstrap required, opened a new log
I20260929 11:28:56.373167 3576 ts_tablet_manager.cc:1397] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260929 11:28:56.375051 3575 tablet_bootstrap.cc:492] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0: No bootstrap required, opened a new log
I20260929 11:28:56.375367 3575 ts_tablet_manager.cc:1397] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.002s
I20260929 11:28:56.383308 3575 raft_consensus.cc:348] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b6bafa329464194b3c265b4afc386b0" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36531 } } peers { permanent_uuid: "a6076fee79e342bc89a4e251883d3b3c" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35561 } } peers { permanent_uuid: "6c6979c4e0374f9fb49345e32bc6ab29" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42213 } }
I20260929 11:28:56.383325 3574 raft_consensus.cc:348] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b6bafa329464194b3c265b4afc386b0" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36531 } } peers { permanent_uuid: "a6076fee79e342bc89a4e251883d3b3c" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35561 } } peers { permanent_uuid: "6c6979c4e0374f9fb49345e32bc6ab29" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42213 } }
I20260929 11:28:56.383783 3575 raft_consensus.cc:374] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:28:56.383919 3574 raft_consensus.cc:374] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:28:56.384090 3574 raft_consensus.cc:729] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6076fee79e342bc89a4e251883d3b3c, State: Initialized, Role: FOLLOWER
I20260929 11:28:56.384055 3575 raft_consensus.cc:729] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b6bafa329464194b3c265b4afc386b0, State: Initialized, Role: FOLLOWER
I20260929 11:28:56.384141 3576 raft_consensus.cc:348] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b6bafa329464194b3c265b4afc386b0" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36531 } } peers { permanent_uuid: "a6076fee79e342bc89a4e251883d3b3c" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35561 } } peers { permanent_uuid: "6c6979c4e0374f9fb49345e32bc6ab29" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42213 } }
I20260929 11:28:56.384599 3576 raft_consensus.cc:374] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:28:56.384694 3576 raft_consensus.cc:729] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c6979c4e0374f9fb49345e32bc6ab29, State: Initialized, Role: FOLLOWER
I20260929 11:28:56.384975 3575 consensus_queue.cc:260] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b6bafa329464194b3c265b4afc386b0" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36531 } } peers { permanent_uuid: "a6076fee79e342bc89a4e251883d3b3c" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35561 } } peers { permanent_uuid: "6c6979c4e0374f9fb49345e32bc6ab29" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42213 } }
I20260929 11:28:56.384956 3574 consensus_queue.cc:260] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b6bafa329464194b3c265b4afc386b0" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36531 } } peers { permanent_uuid: "a6076fee79e342bc89a4e251883d3b3c" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35561 } } peers { permanent_uuid: "6c6979c4e0374f9fb49345e32bc6ab29" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42213 } }
I20260929 11:28:56.385231 3576 consensus_queue.cc:260] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b6bafa329464194b3c265b4afc386b0" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36531 } } peers { permanent_uuid: "a6076fee79e342bc89a4e251883d3b3c" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35561 } } peers { permanent_uuid: "6c6979c4e0374f9fb49345e32bc6ab29" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42213 } }
I20260929 11:28:56.386528 3555 heartbeater.cc:503] Master 127.23.153.62:35795 was elected leader, sending a full tablet report...
I20260929 11:28:56.388415 3576 ts_tablet_manager.cc:1428] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260929 11:28:56.388991 3421 heartbeater.cc:503] Master 127.23.153.62:35795 was elected leader, sending a full tablet report...
I20260929 11:28:56.390719 3574 ts_tablet_manager.cc:1428] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c: Time spent starting tablet: real 0.019s user 0.008s sys 0.008s
I20260929 11:28:56.391635 3575 ts_tablet_manager.cc:1428] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0: Time spent starting tablet: real 0.016s user 0.009s sys 0.005s
I20260929 11:28:56.488263 3581 raft_consensus.cc:482] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:28:56.488560 3581 raft_consensus.cc:504] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b6bafa329464194b3c265b4afc386b0" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36531 } } peers { permanent_uuid: "a6076fee79e342bc89a4e251883d3b3c" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35561 } } peers { permanent_uuid: "6c6979c4e0374f9fb49345e32bc6ab29" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42213 } }
I20260929 11:28:56.490383 3581 leader_election.cc:288] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b6bafa329464194b3c265b4afc386b0 (127.23.153.2:36531), 6c6979c4e0374f9fb49345e32bc6ab29 (127.23.153.3:42213)
W20260929 11:28:56.495461 3556 tablet.cc:2369] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:28:56.496426 3279 tablet.cc:2369] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:28:56.498723 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:56.491240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:53823 (local address 127.23.153.3:42213)
0929 11:28:56.491425 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:56.491430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:56.491719 (+ 289us) server_negotiation.cc:407] Connection header received
0929 11:28:56.491933 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:56.491938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:56.492009 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:56.492115 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:28:56.492869 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.493954 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.496914 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.497743 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.497922 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:56.498013 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:56.498323 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 11:28:56.498460 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:28:56.498972 3584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:56.490772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:42213 (local address 127.23.153.1:53823)
0929 11:28:56.491626 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0929 11:28:56.491640 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 11:28:56.491790 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:28:56.492133 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:28:56.492138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:28:56.492178 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:28:56.492720 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:56.492729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.494101 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.494105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:56.496665 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:56.496686 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.498001 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.498007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:56.498164 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:56.498191 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 11:28:56.498280 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 11:28:56.498457 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":105,"spinlock_wait_cycles":54400,"thread_start_us":472,"threads_started":1}
I20260929 11:28:56.498996 3583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:56.490695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:36531 (local address 127.23.153.1:41523)
0929 11:28:56.491546 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0929 11:28:56.491567 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:28:56.491742 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:28:56.492133 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:28:56.492138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:28:56.492164 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:28:56.492680 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:56.492694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.494099 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.494105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:56.496665 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:28:56.496686 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.498056 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.498062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:28:56.498214 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:56.498238 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 11:28:56.498322 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 11:28:56.498487 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":137,"thread_start_us":722,"threads_started":1}
I20260929 11:28:56.499357 3568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:56.490718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:41523 (local address 127.23.153.2:36531)
0929 11:28:56.490937 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:56.490943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:56.491671 (+ 728us) server_negotiation.cc:407] Connection header received
0929 11:28:56.491925 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:56.491932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:56.492009 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:56.492115 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:28:56.492852 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.493955 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.496900 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.497804 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.497988 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:56.498067 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:56.498408 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 11:28:56.498599 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 11:28:56.500156 3376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7fd58e99764ea38d6d95630776ff9b" candidate_uuid: "a6076fee79e342bc89a4e251883d3b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b6bafa329464194b3c265b4afc386b0" is_pre_election: true
I20260929 11:28:56.500160 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7fd58e99764ea38d6d95630776ff9b" candidate_uuid: "a6076fee79e342bc89a4e251883d3b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c6979c4e0374f9fb49345e32bc6ab29" is_pre_election: true
I20260929 11:28:56.500622 3376 raft_consensus.cc:2454] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6076fee79e342bc89a4e251883d3b3c in term 0.
I20260929 11:28:56.500623 3510 raft_consensus.cc:2454] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6076fee79e342bc89a4e251883d3b3c in term 0.
I20260929 11:28:56.501315 3166 leader_election.cc:302] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c6979c4e0374f9fb49345e32bc6ab29, a6076fee79e342bc89a4e251883d3b3c; no voters:
I20260929 11:28:56.501756 3581 raft_consensus.cc:2781] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:28:56.501927 3581 raft_consensus.cc:482] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:28:56.502054 3581 raft_consensus.cc:3037] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:28:56.504361 3581 raft_consensus.cc:504] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b6bafa329464194b3c265b4afc386b0" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36531 } } peers { permanent_uuid: "a6076fee79e342bc89a4e251883d3b3c" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35561 } } peers { permanent_uuid: "6c6979c4e0374f9fb49345e32bc6ab29" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42213 } }
I20260929 11:28:56.505020 3581 leader_election.cc:288] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [CANDIDATE]: Term 1 election: Requested vote from peers 4b6bafa329464194b3c265b4afc386b0 (127.23.153.2:36531), 6c6979c4e0374f9fb49345e32bc6ab29 (127.23.153.3:42213)
I20260929 11:28:56.505599 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7fd58e99764ea38d6d95630776ff9b" candidate_uuid: "a6076fee79e342bc89a4e251883d3b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c6979c4e0374f9fb49345e32bc6ab29"
I20260929 11:28:56.505606 3376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7fd58e99764ea38d6d95630776ff9b" candidate_uuid: "a6076fee79e342bc89a4e251883d3b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b6bafa329464194b3c265b4afc386b0"
I20260929 11:28:56.505800 3376 raft_consensus.cc:3037] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:28:56.505797 3510 raft_consensus.cc:3037] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:28:56.508148 3510 raft_consensus.cc:2454] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6076fee79e342bc89a4e251883d3b3c in term 1.
I20260929 11:28:56.508224 3376 raft_consensus.cc:2454] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6076fee79e342bc89a4e251883d3b3c in term 1.
I20260929 11:28:56.508704 3166 leader_election.cc:302] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c6979c4e0374f9fb49345e32bc6ab29, a6076fee79e342bc89a4e251883d3b3c; no voters:
I20260929 11:28:56.509055 3581 raft_consensus.cc:2781] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:28:56.509493 3581 raft_consensus.cc:686] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [term 1 LEADER]: Becoming Leader. State: Replica: a6076fee79e342bc89a4e251883d3b3c, State: Running, Role: LEADER
I20260929 11:28:56.509984 3581 consensus_queue.cc:237] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b6bafa329464194b3c265b4afc386b0" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36531 } } peers { permanent_uuid: "a6076fee79e342bc89a4e251883d3b3c" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35561 } } peers { permanent_uuid: "6c6979c4e0374f9fb49345e32bc6ab29" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42213 } }
I20260929 11:28:56.515357 3064 catalog_manager.cc:5495] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c reported cstate change: term changed from 0 to 1, leader changed from <none> to a6076fee79e342bc89a4e251883d3b3c (127.23.153.1). New cstate: current_term: 1 leader_uuid: "a6076fee79e342bc89a4e251883d3b3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b6bafa329464194b3c265b4afc386b0" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6076fee79e342bc89a4e251883d3b3c" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 35561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c6979c4e0374f9fb49345e32bc6ab29" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42213 } health_report { overall_health: UNKNOWN } } }
I20260929 11:28:56.536623 3573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:56.530750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.23.153.1:35561)
0929 11:28:56.530949 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:56.530955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:56.530976 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:28:56.531238 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:56.531242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:56.531334 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:56.531491 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:28:56.531853 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.532601 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.534231 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.534669 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.534793 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:56.535421 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:28:56.535975 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:28:56.536127 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:56.536239 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 11:28:56.536354 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:28:56.545755 3510 raft_consensus.cc:1261] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29 [term 1 FOLLOWER]: Refusing update from remote peer a6076fee79e342bc89a4e251883d3b3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:28:56.545755 3376 raft_consensus.cc:1261] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0 [term 1 FOLLOWER]: Refusing update from remote peer a6076fee79e342bc89a4e251883d3b3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:28:56.546871 3585 consensus_queue.cc:1035] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b6bafa329464194b3c265b4afc386b0" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 36531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:28:56.547317 3581 consensus_queue.cc:1035] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c6979c4e0374f9fb49345e32bc6ab29" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:28:56.558995 3588 mvcc.cc:205] Tried to move back new op lower bound from 7334630754479878144 to 7334630754349350912. Current Snapshot: MvccSnapshot[applied={T|T < 7334630754479878144}]
I20260929 11:28:56.564388 3590 mvcc.cc:205] Tried to move back new op lower bound from 7334630754479878144 to 7334630754349350912. Current Snapshot: MvccSnapshot[applied={T|T < 7334630754479878144}]
I20260929 11:28:56.575839 3064 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:28:56.576184 3064 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:28:56.580382 3064 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43ad644a4cac4e9392cf8b324aac83b2: Sending DeleteTablet for 3 replicas of tablet cb7fd58e99764ea38d6d95630776ff9b
I20260929 11:28:56.581800 3490 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7fd58e99764ea38d6d95630776ff9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:28:56 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:48938
I20260929 11:28:56.582015 3213 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7fd58e99764ea38d6d95630776ff9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:28:56 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:38286
I20260929 11:28:56.582036 3356 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7fd58e99764ea38d6d95630776ff9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:28:56 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:47118
I20260929 11:28:56.584576 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 3145
I20260929 11:28:56.594867 3599 tablet_replica.cc:330] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0: stopping tablet replica
I20260929 11:28:56.595806 3598 tablet_replica.cc:330] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c: stopping tablet replica
I20260929 11:28:56.596374 3598 raft_consensus.cc:2229] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:28:56.597180 3598 raft_consensus.cc:2258] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:28:56.598802 3599 raft_consensus.cc:2229] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:28:56.599375 3599 raft_consensus.cc:2258] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:28:56.601872 3599 ts_tablet_manager.cc:1905] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:28:56.601922 3597 tablet_replica.cc:330] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29: stopping tablet replica
I20260929 11:28:56.602402 3597 raft_consensus.cc:2229] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:28:56.602928 3597 raft_consensus.cc:2258] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:28:56.604894 3598 ts_tablet_manager.cc:1905] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 11:28:56 dist-test-slave-qk05 krb5kdc[2589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:28:56.609234 3599 ts_tablet_manager.cc:1918] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:28:56.609428 3599 log.cc:1196] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/cb7fd58e99764ea38d6d95630776ff9b
I20260929 11:28:56.610087 3599 ts_tablet_manager.cc:1939] T cb7fd58e99764ea38d6d95630776ff9b P 4b6bafa329464194b3c265b4afc386b0: Deleting consensus metadata
I20260929 11:28:56.612253 3050 catalog_manager.cc:4841] TS 4b6bafa329464194b3c265b4afc386b0 (127.23.153.2:36531): tablet cb7fd58e99764ea38d6d95630776ff9b (table test-table [id=c47f2c50b2694ec6894303bacafeec06]) successfully deleted
I20260929 11:28:56.613366 3598 ts_tablet_manager.cc:1918] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:28:56.613571 3598 log.cc:1196] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/cb7fd58e99764ea38d6d95630776ff9b
I20260929 11:28:56.614316 3598 ts_tablet_manager.cc:1939] T cb7fd58e99764ea38d6d95630776ff9b P a6076fee79e342bc89a4e251883d3b3c: Deleting consensus metadata
I20260929 11:28:56.616509 3050 catalog_manager.cc:4841] TS a6076fee79e342bc89a4e251883d3b3c (127.23.153.1:35561): tablet cb7fd58e99764ea38d6d95630776ff9b (table test-table [id=c47f2c50b2694ec6894303bacafeec06]) successfully deleted
I20260929 11:28:56.616786 3597 ts_tablet_manager.cc:1905] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:28:56.619601 3573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:56.585839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38298 (local address 127.23.153.1:35561)
0929 11:28:56.586019 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:56.586026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:56.597296 (+ 11270us) server_negotiation.cc:407] Connection header received
0929 11:28:56.597478 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:56.597483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:56.597581 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:56.597694 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:28:56.599222 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.600714 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.603690 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.604363 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.604492 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:56.610747 (+ 6255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:28:56.610769 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:28:56.610793 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:28:56.610864 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:28:56.615554 (+ 4690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:56.616609 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:56.616619 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:56.616623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:56.616805 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:56.617297 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:56.617302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:56.617306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:56.617804 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:28:56.618119 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:56.618896 (+ 777us) server_negotiation.cc:299] Negotiation successful
0929 11:28:56.619210 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:28:56.621227 3273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:28:56.624670 3597 ts_tablet_manager.cc:1918] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:28:56.624861 3597 log.cc:1196] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/cb7fd58e99764ea38d6d95630776ff9b
I20260929 11:28:56.625387 3597 ts_tablet_manager.cc:1939] T cb7fd58e99764ea38d6d95630776ff9b P 6c6979c4e0374f9fb49345e32bc6ab29: Deleting consensus metadata
I20260929 11:28:56.627410 3049 catalog_manager.cc:4841] TS 6c6979c4e0374f9fb49345e32bc6ab29 (127.23.153.3:42213): tablet cb7fd58e99764ea38d6d95630776ff9b (table test-table [id=c47f2c50b2694ec6894303bacafeec06]) successfully deleted
I20260929 11:28:56.726779 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 3145
I20260929 11:28:56.743376 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 3284
Sep 29 11:28:56 dist-test-slave-qk05 krb5kdc[2589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.2@KRBTEST.COM
I20260929 11:28:56.760401 3568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:56.744357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47124 (local address 127.23.153.2:36531)
0929 11:28:56.744531 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:56.744535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:56.744556 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:28:56.744676 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:56.744680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:56.744750 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:56.744831 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:28:56.746303 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.747182 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.748541 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.748955 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.749067 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:56.753814 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:28:56.753832 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:28:56.753850 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:28:56.753897 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:28:56.756915 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:56.757771 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:56.757779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:56.757781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:56.757924 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:56.758629 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:56.758634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:56.758636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:56.759118 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:28:56.759356 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:56.759861 (+ 505us) server_negotiation.cc:299] Negotiation successful
0929 11:28:56.760046 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:28:56.761109 3416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:28:56.854085 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 3284
I20260929 11:28:56.868063 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 3427
Sep 29 11:28:56 dist-test-slave-qk05 krb5kdc[2589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.3@KRBTEST.COM
I20260929 11:28:56.885449 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:28:56.868899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48948 (local address 127.23.153.3:42213)
0929 11:28:56.869079 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 11:28:56.869084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:28:56.869104 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:28:56.869244 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:28:56.869248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:28:56.869456 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:28:56.869724 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:28:56.871102 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.871896 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.873356 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:28:56.873888 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:28:56.874006 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:28:56.879199 (+ 5193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:28:56.879218 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:28:56.879235 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:28:56.879279 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:28:56.882161 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:56.883207 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:56.883215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:56.883217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:56.883367 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:28:56.883865 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:28:56.883869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:28:56.883871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:28:56.884390 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:28:56.884647 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 11:28:56.884962 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 11:28:56.885167 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:28:56.886176 3550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:28:56.977283 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 3427
I20260929 11:28:56.991250 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 3033
I20260929 11:28:56.992488 3095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:28:57.107035 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 3033
I20260929 11:28:57.126593 24164 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 11:28:57.463315 24164 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 11:28:57.463549 24164 mini_ranger.cc:66] Stopping Ranger...
I20260929 11:28:57.810333 24164 mini_ranger.cc:68] Stopped Ranger
2026-09-29 11:28:57.810 UTC [2619] LOG: received smart shutdown request
2026-09-29 11:28:57.813 UTC [2619] LOG: background worker "logical replication launcher" (PID 2628) exited with exit code 1
2026-09-29 11:28:57.814 UTC [2623] LOG: shutting down
2026-09-29 11:28:57.957 UTC [2619] LOG: database system is shut down
2026-09-29T11:28:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (55310 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:28:58 dist-test-slave-qk05 krb5kdc[3613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:28:58 dist-test-slave-qk05 krb5kdc[3613](info): set up 2 sockets
Sep 29 11:28:58 dist-test-slave-qk05 krb5kdc[3613](info): commencing operation
krb5kdc: starting...
W20260929 11:28:59.127422 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.010s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:28:59 dist-test-slave-qk05 krb5kdc[3613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:28:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:28:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.153.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.153.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.153.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/rangeradmin_127.23.153.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.153.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/rangeradmin_127.23.153.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.153.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.153.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.153.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/rangerlookup_127.23.153.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.153.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/rangerlookup_127.23.153.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.153.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.153.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.153.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/HTTP_127.23.153.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.153.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/HTTP_127.23.153.20@KRBTEST.COM.keytab.
I20260929 11:28:59.458904 24164 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBSjPgi/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/postgres -l logfile start
2026-09-29 11:29:02.090 UTC [3643] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 11:29:02.091 UTC [3643] LOG: listening on IPv4 address "127.23.153.20", port 38085
2026-09-29 11:29:02.094 UTC [3643] LOG: listening on Unix socket "/tmp/.s.PGSQL.38085"
2026-09-29 11:29:02.118 UTC [3646] LOG: database system was shut down at 2026-09-29 11:29:00 UTC
2026-09-29 11:29:02.122 UTC [3643] LOG: database system is ready to accept connections
I20260929 11:29:03.035480 24164 mini_postgres.cc:96] Postgres bound to 38085
2026-09-29 11:29:03.052 UTC [3654] FATAL: database "slave" does not exist
127.23.153.20:38085 - accepting connections
I20260929 11:29:03.052815 24164 mini_ranger.cc:161] Starting Ranger...
I20260929 11:29:03.080204 24164 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 11:29:03.592999 24164 mini_ranger.cc:87] Created ranger Postgres database
I20260929 11:29:03.593125 24164 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-admin
2026-09-29 11:29:03,900 [I] DB FLAVOR :POSTGRES
2026-09-29 11:29:03,901 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 11:29:03,901 [I] Checking connection..
2026-09-29 11:29:03,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 11:29:04,187 [I] Checking connection passed.
2026-09-29 11:29:04,187 [I] --------- Verifying version history table ---------
2026-09-29 11:29:04,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 11:29:04,467 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 11:29:04,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 11:29:04,728 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 11:29:04,728 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 11:29:04,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 11:29:04.997 UTC [3718] WARNING: there is no transaction in progress
2026-09-29 11:29:05,003 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 11:29:05,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 11:29:05,281 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 11:29:05,281 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 11:29:05,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 11:29:05,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 11:29:05,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qk05.c.gcp-upstream.internal','N') ;"
2026-09-29 11:29:06,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 11:29:06,364 [I] Table x_portal_user does not exist in database ranger
2026-09-29 11:29:06,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 11:29:06,645 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 11:29:06,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 11:29:06,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 11:29:07,178 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 11:29:07,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 11:29:08.291 UTC [3826] WARNING: there is no transaction in progress
2026-09-29 11:29:08.308 UTC [3826] WARNING: there is no transaction in progress
2026-09-29 11:29:08.325 UTC [3826] WARNING: there is no transaction in progress
2026-09-29 11:29:08.342 UTC [3826] WARNING: there is no transaction in progress
2026-09-29 11:29:08.361 UTC [3826] WARNING: there is no transaction in progress
2026-09-29 11:29:08.381 UTC [3826] WARNING: there is no transaction in progress
2026-09-29 11:29:08.453 UTC [3826] WARNING: there is no transaction in progress
2026-09-29 11:29:08.461 UTC [3826] WARNING: there is no transaction in progress
2026-09-29 11:29:08.477 UTC [3826] WARNING: there is no transaction in progress
2026-09-29 11:29:08.489 UTC [3826] WARNING: there is no transaction in progress
2026-09-29 11:29:08.500 UTC [3826] WARNING: there is no transaction in progress
2026-09-29 11:29:08.514 UTC [3826] WARNING: there is no transaction in progress
2026-09-29 11:29:08.525 UTC [3826] WARNING: there is no transaction in progress
2026-09-29 11:29:08.532 UTC [3826] WARNING: there is no transaction in progress
2026-09-29 11:29:08.540 UTC [3826] WARNING: there is no transaction in progress
2026-09-29 11:29:08.547 UTC [3826] WARNING: there is no transaction in progress
2026-09-29 11:29:09,306 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 11:29:09,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 11:29:09,582 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 11:29:09,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 11:29:09,857 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 11:29:09,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 11:29:10,146 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 11:29:10,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 11:29:10,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 11:29:10,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qk05.c.gcp-upstream.internal';"
2026-09-29 11:29:10,958 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 11:29:10,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 11:29:11,239 [I] DB_PATCHES have already been applied
I20260929 11:29:11.246644 24164 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBSjPgi/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 11:29:11.246768 24164 mini_ranger.cc:193] Using host: 127.23.153.20
I20260929 11:29:11.262274 24164 mini_ranger.cc:227] Ranger admin URL: http://127.23.153.20:36523
Sep 29, 2026 11:29:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 11:29:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 11:29:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 11:29:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 11:29:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 11:29:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 11:29:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 11:29:12.321347 24164 mini_ranger.cc:160] Time spent starting Ranger: real 9.269s user 0.003s sys 0.040s
Sep 29, 2026 11:29:35 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 11:29:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 11:29:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 11:29:36 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 11:29:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 11:29:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 11:29:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 11:29:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 11:29:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 11:29:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 11:29:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 11:29:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 11:29:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 11:29:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 11:29:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 11:29:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 11:29:36 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 11:29:37 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 11:29:37.773890 24164 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.153.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.153.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.153.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/rangerkms_127.23.153.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.153.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/rangerkms_127.23.153.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.153.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.153.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.153.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/HTTP_127.23.153.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.153.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/HTTP_127.23.153.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 11:29:37 dist-test-slave-qk05 krb5kdc[3613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681377, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 11:29:37.989463 24164 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 11:29:38.012821 24164 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 11:29:39.045990 24164 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 11:29:39.046094 24164 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-kms
I20260929 11:29:39.046147 24164 mini_ranger_kms.cc:227] Using postgres at 127.23.153.20:38085
2026-09-29 11:29:39,415 [I] DB FLAVOR :POSTGRES
2026-09-29 11:29:39,416 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 11:29:39,416 [I] Checking connection
2026-09-29 11:29:39,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 11:29:39,697 [I] connection success
2026-09-29 11:29:39,698 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 11:29:39,698 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 11:29:39,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 11:29:39,988 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 11:29:39,989 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 11:29:39,989 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 11:29:39,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.153.20:38085/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 11:29:40,316 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 11:29:40.950256 24164 mini_ranger_kms.cc:310] Created kms service
I20260929 11:29:41.154304 24164 mini_ranger_kms.cc:326] Created kudu user
I20260929 11:29:41.204511 24164 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 11:29:41 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681377, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
I20260929 11:29:41.706615 24164 mini_ranger_kms.cc:384] Added ranger policy
I20260929 11:29:41.706984 24164 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBSjPgi/build/asan/bin/postgresql.jar:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBSjPgi/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 11:29:41.707082 24164 mini_ranger_kms.cc:240] Using host: 127.23.153.20
I20260929 11:29:41.722762 24164 mini_ranger_kms.cc:276] Ranger KMS PID: 4007
I20260929 11:29:41.722898 24164 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.153.20:58279
Sep 29, 2026 11:29:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 11:29:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 11:29:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 11:29:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 11:29:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 11:29:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 11:29:43.778935 24164 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.789s user 0.066s sys 0.318s
I20260929 11:29:43.779233 24164 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 11:29:43.779426 24164 mini_ranger_kms.cc:401] 127.23.153.20:58279/kms/v1/keys
Sep 29 11:29:44 dist-test-slave-qk05 krb5kdc[3613](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790681384, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.153.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 11:29:44 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681384, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.153.20@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
Sep 29 11:29:45 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681384, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.153.20@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
Sep 29 11:29:47 dist-test-slave-qk05 krb5kdc[3613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:29:47.871621 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:33971
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:33971
--ranger_config_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.153.20:58279/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:29:48.246541 4057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:29:48.246977 4057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:29:48.247164 4057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:29:48.256177 4057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:29:48.256340 4057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:29:48.256409 4057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:29:48.256495 4057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:29:48.256564 4057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:29:48.267625 4057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.153.20:58279/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:33971
--ranger_config_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:33971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:29:48.269241 4057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:29:48.270731 4057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:29:48.284083 4062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:48.284116 4065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:48.284150 4063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:29:48.285317 4057 server_base.cc:1034] running on GCE node
I20260929 11:29:48.286379 4057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:29:48.288100 4057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:29:48.289383 4057 hybrid_clock.cc:648] HybridClock initialized: now 1790681388289331 us; error 41 us; skew 500 ppm
I20260929 11:29:48.289826 4057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:29:48 dist-test-slave-qk05 krb5kdc[3613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681388, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:29:48.293486 4057 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:29:48.295351 4057 webserver.cc:460] Webserver started at http://127.23.153.62:42277/ using document root <none> and password file <none>
I20260929 11:29:48.296221 4057 fs_manager.cc:359] Metadata directory not provided
I20260929 11:29:48.296375 4057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:29:48.296741 4057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:29:48 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681388, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
I20260929 11:29:48.364327 4057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "ddfcabeb084a46be92ac593ebfdfeea8"
format_stamp: "Formatted at 2026-09-29 11:29:48 on dist-test-slave-qk05"
server_key: "e71063d9c22e7c015a9613d4a8ba9de3"
server_key_iv: "b203268b027a5f1b224e330b47349aa9"
server_key_version: "kuduclusterkey@0"
I20260929 11:29:48.365628 4057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "ddfcabeb084a46be92ac593ebfdfeea8"
format_stamp: "Formatted at 2026-09-29 11:29:48 on dist-test-slave-qk05"
server_key: "e71063d9c22e7c015a9613d4a8ba9de3"
server_key_iv: "b203268b027a5f1b224e330b47349aa9"
server_key_version: "kuduclusterkey@0"
I20260929 11:29:48.372478 4057 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 11:29:48.396832 4072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:48.399708 4057 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.001s
I20260929 11:29:48.399991 4057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "ddfcabeb084a46be92ac593ebfdfeea8"
format_stamp: "Formatted at 2026-09-29 11:29:48 on dist-test-slave-qk05"
server_key: "e71063d9c22e7c015a9613d4a8ba9de3"
server_key_iv: "b203268b027a5f1b224e330b47349aa9"
server_key_version: "kuduclusterkey@0"
I20260929 11:29:48.400524 4057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:29:48.415614 4057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:29:48.421101 4057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:29:48.421474 4057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:29:48.441532 4057 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:33971
I20260929 11:29:48.441576 4123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:33971 every 8 connection(s)
I20260929 11:29:48.443053 4057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:29:48.447505 4124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:29:48.449908 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 4057
I20260929 11:29:48.450119 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:29:48.450800 24164 external_mini_cluster.cc:1428] Setting key cd3a49f3e804562b70bc39fe8290b7c9
I20260929 11:29:48.461848 4124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8: Bootstrap starting.
Sep 29 11:29:48 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:29:48.467563 4124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8: Neither blocks nor log segments found. Creating new log.
I20260929 11:29:48.468926 4124 log.cc:824] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8: Log is configured to *not* fsync() on all Append() calls
I20260929 11:29:48.472476 4124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8: No bootstrap required, opened a new log
I20260929 11:29:48.477337 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:48.453234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34344 (local address 127.23.153.62:33971)
0929 11:29:48.454171 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:48.454186 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:48.454235 (+ 49us) server_negotiation.cc:407] Connection header received
0929 11:29:48.456139 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:48.456169 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:48.456661 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:48.457109 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:48.458663 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:48.460184 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:48.461625 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:48.462496 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:48.462712 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:48.467797 (+ 5085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:48.467808 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:48.467828 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:48.467883 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:48.472185 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:48.473132 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:48.473140 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:48.473148 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:48.473353 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:48.473790 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:48.473794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:48.473796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:48.474533 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:48.474896 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:48.475446 (+ 550us) server_negotiation.cc:299] Negotiation successful
0929 11:29:48.475949 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":427,"threads_started":1}
I20260929 11:29:48.483742 4124 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfcabeb084a46be92ac593ebfdfeea8" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33971 } }
I20260929 11:29:48.484328 4124 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:29:48.484529 4124 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfcabeb084a46be92ac593ebfdfeea8, State: Initialized, Role: FOLLOWER
I20260929 11:29:48.485424 4124 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfcabeb084a46be92ac593ebfdfeea8" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33971 } }
I20260929 11:29:48.485773 4124 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:29:48.485930 4124 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:29:48.486131 4124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:29:48.489261 4124 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfcabeb084a46be92ac593ebfdfeea8" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33971 } }
I20260929 11:29:48.489986 4124 leader_election.cc:302] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddfcabeb084a46be92ac593ebfdfeea8; no voters:
I20260929 11:29:48.490650 4124 leader_election.cc:288] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:29:48.490804 4129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:29:48.491572 4129 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [term 1 LEADER]: Becoming Leader. State: Replica: ddfcabeb084a46be92ac593ebfdfeea8, State: Running, Role: LEADER
I20260929 11:29:48.492192 4129 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfcabeb084a46be92ac593ebfdfeea8" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33971 } }
I20260929 11:29:48.492962 4124 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:29:48.495363 4131 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddfcabeb084a46be92ac593ebfdfeea8. Latest consensus state: current_term: 1 leader_uuid: "ddfcabeb084a46be92ac593ebfdfeea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfcabeb084a46be92ac593ebfdfeea8" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33971 } } }
I20260929 11:29:48.495868 4131 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [sys.catalog]: This master's current role is: LEADER
I20260929 11:29:48.496654 4130 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddfcabeb084a46be92ac593ebfdfeea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfcabeb084a46be92ac593ebfdfeea8" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33971 } } }
I20260929 11:29:48.496986 4130 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [sys.catalog]: This master's current role is: LEADER
I20260929 11:29:48.499399 4138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:29:48.500000 4124 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 11:29:48.512686 4138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:29:48.527869 4138 catalog_manager.cc:1347] Generated new cluster ID: ea6cccc86ccc441797ef143af695b636
I20260929 11:29:48.528050 4138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:29:48.548061 4138 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:29:48.550278 4138 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:29:48.569237 4138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8: Generated new TSK 0
I20260929 11:29:48.570498 4138 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
I20260929 11:29:49.365031 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.153.20:58279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:33971
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:29:49 dist-test-slave-qk05 krb5kdc[3613](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790681389, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 11:29:49.875329 4169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:29:49.875836 4169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:29:49.876021 4169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:29:49.888397 4169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:29:49.888628 4169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:29:49.888855 4169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:29:49.907099 4169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.153.20:58279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:33971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:29:49.908888 4169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:29:49.910683 4169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:29:49.940289 4179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:49.945420 4180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:49.954291 4182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:29:49.955344 4169 server_base.cc:1034] running on GCE node
I20260929 11:29:49.956259 4169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:29:49.960114 4169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:29:49.961664 4169 hybrid_clock.cc:648] HybridClock initialized: now 1790681389961500 us; error 128 us; skew 500 ppm
I20260929 11:29:49.962471 4169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:29:49 dist-test-slave-qk05 krb5kdc[3613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681389, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:29:49.971918 4169 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:29:49.978555 4169 webserver.cc:460] Webserver started at http://127.23.153.1:34023/ using document root <none> and password file <none>
I20260929 11:29:49.979557 4169 fs_manager.cc:359] Metadata directory not provided
I20260929 11:29:49.979703 4169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:29:49.980079 4169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:29:49 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681389, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
I20260929 11:29:50.002828 4169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "be6b7facd77445e59039bb2104b7be19"
format_stamp: "Formatted at 2026-09-29 11:29:49 on dist-test-slave-qk05"
server_key: "ca311c02bd4e83145ab239ef2d8bd470"
server_key_iv: "742a9080dfe5f3f252f9b25f4450fbd3"
server_key_version: "kuduclusterkey@0"
I20260929 11:29:50.005028 4169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "be6b7facd77445e59039bb2104b7be19"
format_stamp: "Formatted at 2026-09-29 11:29:49 on dist-test-slave-qk05"
server_key: "ca311c02bd4e83145ab239ef2d8bd470"
server_key_iv: "742a9080dfe5f3f252f9b25f4450fbd3"
server_key_version: "kuduclusterkey@0"
I20260929 11:29:50.014890 4169 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s
I20260929 11:29:50.053176 4189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:50.055490 4169 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.006s sys 0.012s
I20260929 11:29:50.055831 4169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "be6b7facd77445e59039bb2104b7be19"
format_stamp: "Formatted at 2026-09-29 11:29:49 on dist-test-slave-qk05"
server_key: "ca311c02bd4e83145ab239ef2d8bd470"
server_key_iv: "742a9080dfe5f3f252f9b25f4450fbd3"
server_key_version: "kuduclusterkey@0"
I20260929 11:29:50.056330 4169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:29:50.094439 4169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:29:50.098836 4169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:29:50.099340 4169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:29:50.101380 4169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:29:50.105031 4169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:29:50.105247 4169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 11:29:50.105407 4169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:29:50.105717 4169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:50.169346 4169 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:45065
I20260929 11:29:50.169365 4301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:45065 every 8 connection(s)
I20260929 11:29:50.171553 4169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:29:50.177027 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 4169
I20260929 11:29:50.177394 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:29:50.178457 24164 external_mini_cluster.cc:1428] Setting key e01b36289764a93e709813c507a1fe5a
Sep 29 11:29:50 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681389, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:29:50.225396 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:50.174955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:42097 (local address 127.23.153.62:33971)
0929 11:29:50.192126 (+ 17171us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:50.192136 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:50.194139 (+ 2003us) server_negotiation.cc:407] Connection header received
0929 11:29:50.199387 (+ 5248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:50.199393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:50.199527 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:50.199641 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:50.203491 (+ 3850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:50.204703 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:50.207880 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:50.208592 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:50.208718 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:50.215997 (+ 7279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:50.216042 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:50.216048 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:50.216107 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:50.220295 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:50.221659 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:50.221666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:50.221670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:50.221867 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:50.222687 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:50.222693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:50.222697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:50.223236 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:50.223388 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:50.224692 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0929 11:29:50.224978 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16988,"thread_start_us":18986,"threads_started":1}
I20260929 11:29:50.228366 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:50.175768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:33971 (local address 127.23.153.1:42097)
0929 11:29:50.192810 (+ 17042us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:50.192882 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 11:29:50.198678 (+ 5796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:29:50.201411 (+ 2733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:29:50.201429 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:29:50.202155 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:29:50.203169 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:50.203208 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:50.204986 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:50.204992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:50.207604 (+ 2612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:50.207635 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:50.209182 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:50.209191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:50.209565 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:50.210486 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:29:50.210528 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:29:50.215582 (+ 5054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:29:50.220574 (+ 4992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:50.220587 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:50.220609 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:50.221448 (+ 839us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:50.222041 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:50.222046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:50.222050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:50.222513 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:50.223502 (+ 989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:29:50.223510 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:29:50.224120 (+ 610us) client_negotiation.cc:769] Sending connection context
0929 11:29:50.225983 (+ 1863us) client_negotiation.cc:240] Negotiation successful
0929 11:29:50.226661 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16697,"thread_start_us":16425,"threads_started":1}
I20260929 11:29:50.230708 4302 heartbeater.cc:347] Connected to a master server at 127.23.153.62:33971
I20260929 11:29:50.231199 4302 heartbeater.cc:464] Registering TS with master...
I20260929 11:29:50.232174 4302 heartbeater.cc:511] Master 127.23.153.62:33971 requested a full tablet report, sending...
I20260929 11:29:50.235188 4089 ts_manager.cc:194] Registered new tserver with Master: be6b7facd77445e59039bb2104b7be19 (127.23.153.1:45065)
I20260929 11:29:50.237360 4089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:42097
WARNING: no policy specified for kudu/127.23.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.2@KRBTEST.COM" created.
Sep 29 11:29:50 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681389, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
I20260929 11:29:50.277076 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.153.20:58279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:33971
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:29:50 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681389, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
W20260929 11:29:50.813970 4308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:29:50.814539 4308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:29:50.814749 4308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:29:50.826211 4308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:29:50.826500 4308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:29:50.826754 4308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:29:50.841600 4308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.153.20:58279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:33971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:29:50.844367 4308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:29:50.846633 4308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:29:50.865803 4316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:50.871543 4315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:50.874094 4318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:29:50.875480 4308 server_base.cc:1034] running on GCE node
I20260929 11:29:50.876247 4308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:29:50.877828 4308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:29:50.883420 4308 hybrid_clock.cc:648] HybridClock initialized: now 1790681390883173 us; error 234 us; skew 500 ppm
I20260929 11:29:50.884563 4308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:29:50 dist-test-slave-qk05 krb5kdc[3613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681390, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:29:50.889904 4308 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:29:50.903663 4308 webserver.cc:460] Webserver started at http://127.23.153.2:46385/ using document root <none> and password file <none>
I20260929 11:29:50.904953 4308 fs_manager.cc:359] Metadata directory not provided
I20260929 11:29:50.905216 4308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:29:50.905759 4308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:29:50 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681390, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
I20260929 11:29:50.924683 4308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "dfb424c2a3b741bc985bf0a9be51d018"
format_stamp: "Formatted at 2026-09-29 11:29:50 on dist-test-slave-qk05"
server_key: "17c170969db065d1e3dff1f92a67ddf7"
server_key_iv: "dc3e9fc93597ec82334c2872e5914d52"
server_key_version: "kuduclusterkey@0"
I20260929 11:29:50.925670 4308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "dfb424c2a3b741bc985bf0a9be51d018"
format_stamp: "Formatted at 2026-09-29 11:29:50 on dist-test-slave-qk05"
server_key: "17c170969db065d1e3dff1f92a67ddf7"
server_key_iv: "dc3e9fc93597ec82334c2872e5914d52"
server_key_version: "kuduclusterkey@0"
I20260929 11:29:50.932149 4308 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 11:29:50.955310 4332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:50.956959 4308 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.001s
I20260929 11:29:50.957175 4308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "dfb424c2a3b741bc985bf0a9be51d018"
format_stamp: "Formatted at 2026-09-29 11:29:50 on dist-test-slave-qk05"
server_key: "17c170969db065d1e3dff1f92a67ddf7"
server_key_iv: "dc3e9fc93597ec82334c2872e5914d52"
server_key_version: "kuduclusterkey@0"
I20260929 11:29:50.957402 4308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:29:50.974951 4308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:29:50.980233 4308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:29:50.980607 4308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:29:50.983245 4308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:29:50.985271 4308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:29:50.985383 4308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:50.985518 4308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:29:50.985576 4308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:51.034389 4308 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:44293
I20260929 11:29:51.034420 4444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:44293 every 8 connection(s)
I20260929 11:29:51.036181 4308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:29:51.046633 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 4308
I20260929 11:29:51.046854 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:29:51.047425 24164 external_mini_cluster.cc:1428] Setting key 3deb5abcb79a4ffbc9f5dbd3004df7dd
Sep 29 11:29:51 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681390, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.153.3@KRBTEST.COM; defaulting to no policy
I20260929 11:29:51.090525 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:51.039336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:59135 (local address 127.23.153.62:33971)
0929 11:29:51.067574 (+ 28238us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:51.067581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:51.067616 (+ 35us) server_negotiation.cc:407] Connection header received
0929 11:29:51.068700 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:51.068706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:51.068804 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:51.071224 (+ 2420us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:51.071416 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:51.072525 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:51.075297 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:51.075942 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:51.076099 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:51.081734 (+ 5635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:51.081743 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:51.081748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:51.081801 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:51.085697 (+ 3896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:51.086866 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:51.086872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:51.086876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:51.087106 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:51.087693 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:51.087698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:51.087702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:51.088098 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:51.088258 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:51.089728 (+ 1470us) server_negotiation.cc:299] Negotiation successful
0929 11:29:51.090040 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28054,"thread_start_us":28278,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.3@KRBTEST.COM" created.
I20260929 11:29:51.093102 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:51.040852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:33971 (local address 127.23.153.2:59135)
0929 11:29:51.064537 (+ 23685us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:51.064602 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 11:29:51.068180 (+ 3578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:29:51.069281 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:29:51.069295 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:29:51.069978 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:29:51.070921 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:51.070953 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:51.072809 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:51.072816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:51.074977 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:51.075001 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:51.076466 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:51.076474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:51.076820 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:51.078135 (+ 1315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:29:51.078169 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:29:51.081341 (+ 3172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:29:51.085963 (+ 4622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:51.085974 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:51.085997 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:51.086674 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:51.087259 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:51.087264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:51.087267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:51.087547 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:51.088346 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:29:51.088354 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:29:51.089048 (+ 694us) client_negotiation.cc:769] Sending connection context
0929 11:29:51.089406 (+ 358us) client_negotiation.cc:240] Negotiation successful
0929 11:29:51.091502 (+ 2096us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23286,"thread_start_us":26036,"threads_started":1}
I20260929 11:29:51.095597 4445 heartbeater.cc:347] Connected to a master server at 127.23.153.62:33971
I20260929 11:29:51.095975 4445 heartbeater.cc:464] Registering TS with master...
I20260929 11:29:51.096959 4445 heartbeater.cc:511] Master 127.23.153.62:33971 requested a full tablet report, sending...
I20260929 11:29:51.099515 4089 ts_manager.cc:194] Registered new tserver with Master: dfb424c2a3b741bc985bf0a9be51d018 (127.23.153.2:44293)
I20260929 11:29:51.100618 4089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:59135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
I20260929 11:29:51.130918 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.153.20:58279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:33971
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:29:51.241843 4302 heartbeater.cc:503] Master 127.23.153.62:33971 was elected leader, sending a full tablet report...
W20260929 11:29:51.609424 4451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:29:51.609763 4451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:29:51.609894 4451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:29:51.618116 4451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:29:51.618278 4451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:29:51.618455 4451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:29:51.629551 4451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.153.20:58279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:33971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:29:51.631088 4451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:29:51.632501 4451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:29:51.649632 4456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:51.649719 4457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:51.649641 4459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:29:51.650727 4451 server_base.cc:1034] running on GCE node
I20260929 11:29:51.651496 4451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:29:51.652645 4451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:29:51.653896 4451 hybrid_clock.cc:648] HybridClock initialized: now 1790681391653837 us; error 66 us; skew 500 ppm
I20260929 11:29:51.654289 4451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:29:51 dist-test-slave-qk05 krb5kdc[3613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681391, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:29:51.658484 4451 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:29:51.661190 4451 webserver.cc:460] Webserver started at http://127.23.153.3:39623/ using document root <none> and password file <none>
I20260929 11:29:51.662251 4451 fs_manager.cc:359] Metadata directory not provided
I20260929 11:29:51.662446 4451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:29:51.662806 4451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 11:29:51 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681391, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
I20260929 11:29:51.682683 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "8877ed09137d496c9284dd61c0d223d1"
format_stamp: "Formatted at 2026-09-29 11:29:51 on dist-test-slave-qk05"
server_key: "5a16a755266ba1cf15727ea8c1fc1450"
server_key_iv: "371bfeb5898bc18f543c9c12339c9460"
server_key_version: "kuduclusterkey@0"
I20260929 11:29:51.683835 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "8877ed09137d496c9284dd61c0d223d1"
format_stamp: "Formatted at 2026-09-29 11:29:51 on dist-test-slave-qk05"
server_key: "5a16a755266ba1cf15727ea8c1fc1450"
server_key_iv: "371bfeb5898bc18f543c9c12339c9460"
server_key_version: "kuduclusterkey@0"
I20260929 11:29:51.689867 4451 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:29:51.709534 4466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:51.711874 4451 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 11:29:51.712122 4451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "8877ed09137d496c9284dd61c0d223d1"
format_stamp: "Formatted at 2026-09-29 11:29:51 on dist-test-slave-qk05"
server_key: "5a16a755266ba1cf15727ea8c1fc1450"
server_key_iv: "371bfeb5898bc18f543c9c12339c9460"
server_key_version: "kuduclusterkey@0"
I20260929 11:29:51.712416 4451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:29:51.732734 4451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:29:51.736904 4451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:29:51.737195 4451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:29:51.738582 4451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:29:51.740535 4451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:29:51.740617 4451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:51.740732 4451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:29:51.740778 4451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:51.775346 4451 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:36957
I20260929 11:29:51.775379 4578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:36957 every 8 connection(s)
I20260929 11:29:51.776786 4451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:29:51.779176 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 4451
I20260929 11:29:51.779372 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:29:51.780000 24164 external_mini_cluster.cc:1428] Setting key 703c8d7f0c418be53f585482ebd63e7a
Sep 29 11:29:51 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681391, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:29:51.800885 4581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:51.779754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:60847 (local address 127.23.153.62:33971)
0929 11:29:51.780432 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:51.780437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:51.782270 (+ 1833us) server_negotiation.cc:407] Connection header received
0929 11:29:51.784641 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:51.784646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:51.784745 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:51.784851 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:51.786496 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:51.787316 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:51.789171 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:51.789629 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:51.789746 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:51.793552 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:51.793560 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:51.793564 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:51.793611 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:51.796945 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:51.797799 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:51.797805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:51.797808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:51.797947 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:51.798417 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:51.798421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:51.798423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:51.798729 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:51.798870 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:51.800191 (+ 1321us) server_negotiation.cc:299] Negotiation successful
0929 11:29:51.800496 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":260,"threads_started":1}
I20260929 11:29:51.802008 4582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:51.780526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:33971 (local address 127.23.153.3:60847)
0929 11:29:51.781975 (+ 1449us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:51.782036 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 11:29:51.784286 (+ 2250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:29:51.785111 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:29:51.785120 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:29:51.785585 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:29:51.786272 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:51.786293 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:51.787579 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:51.787584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:51.788937 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:51.788954 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:51.789897 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:51.789903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:51.790172 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:51.790927 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:29:51.790955 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:29:51.793179 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:29:51.797124 (+ 3945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:51.797135 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:51.797152 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:51.797666 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:51.798062 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:51.798068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:51.798072 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:51.798295 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:51.798863 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:29:51.798872 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:29:51.799582 (+ 710us) client_negotiation.cc:769] Sending connection context
0929 11:29:51.800005 (+ 423us) client_negotiation.cc:240] Negotiation successful
0929 11:29:51.800741 (+ 736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"thread_start_us":703,"threads_started":1}
I20260929 11:29:51.803565 4579 heartbeater.cc:347] Connected to a master server at 127.23.153.62:33971
I20260929 11:29:51.803894 4579 heartbeater.cc:464] Registering TS with master...
I20260929 11:29:51.804646 4579 heartbeater.cc:511] Master 127.23.153.62:33971 requested a full tablet report, sending...
I20260929 11:29:51.806249 4089 ts_manager.cc:194] Registered new tserver with Master: 8877ed09137d496c9284dd61c0d223d1 (127.23.153.3:36957)
I20260929 11:29:51.807417 4089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:60847
I20260929 11:29:51.809267 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:29:51.811477 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 4169
Sep 29 11:29:51 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:29:51.830276 4584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:51.812671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.23.153.1:45065)
0929 11:29:51.813310 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:51.813317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:51.813344 (+ 27us) server_negotiation.cc:407] Connection header received
0929 11:29:51.813407 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:51.813414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:51.813644 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:51.813776 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:51.815331 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:51.816561 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:51.818232 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:51.818853 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:51.818987 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:51.823894 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:51.823911 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:51.823930 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:51.823976 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:51.827415 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:51.828303 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:51.828311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:51.828314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:51.828489 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:51.828827 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:51.828830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:51.828832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:51.829208 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:51.829524 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:51.829763 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 11:29:51.829993 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":266,"threads_started":1}
I20260929 11:29:51.831805 4297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:29:51.918937 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 4169
I20260929 11:29:51.935349 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 4308
Sep 29 11:29:51 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.2@KRBTEST.COM
I20260929 11:29:51.953420 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:51.936328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.23.153.2:44293)
0929 11:29:51.936826 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:51.936834 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:51.936858 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:29:51.936917 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:51.936922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:51.937149 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:51.937297 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:51.938546 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:51.939639 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:51.941051 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:51.941655 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:51.941807 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:51.946784 (+ 4977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:51.946802 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:51.946819 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:51.946871 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:51.950254 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:51.951161 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:51.951169 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:51.951172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:51.951330 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:51.951717 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:51.951721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:51.951723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:51.952104 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:51.952392 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:51.952749 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 11:29:51.953041 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":244,"threads_started":1}
I20260929 11:29:51.954691 4440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:29:52.044272 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 4308
I20260929 11:29:52.064163 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 4451
Sep 29 11:29:52 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.3@KRBTEST.COM
I20260929 11:29:52.085522 4588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:52.065655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32918 (local address 127.23.153.3:36957)
0929 11:29:52.066375 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:52.066383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:52.066421 (+ 38us) server_negotiation.cc:407] Connection header received
0929 11:29:52.066516 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:52.066524 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:52.067075 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:52.067303 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:52.069329 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:52.070397 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:52.071899 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:52.072498 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:52.072649 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:52.077930 (+ 5281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:52.077949 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:52.077967 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:52.078013 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:52.081922 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:52.082921 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:52.082929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:52.082931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:52.083313 (+ 382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:52.083730 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:52.083733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:52.083736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:52.084150 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:52.084471 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:52.084840 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 11:29:52.085151 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":359,"threads_started":1}
I20260929 11:29:52.087762 4574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:29:52.178870 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 4451
I20260929 11:29:52.193481 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 4057
I20260929 11:29:52.194782 4119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:29:52.305058 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 4057
I20260929 11:29:52.323529 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:33971
--webserver_interface=127.23.153.62
--webserver_port=42277
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:33971
--ranger_config_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.153.20:58279/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.153.20:58279/kms with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:29:52.678504 4591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:29:52.678867 4591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:29:52.678988 4591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:29:52.688926 4591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:29:52.689148 4591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:29:52.689407 4591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:29:52.689555 4591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:29:52.689625 4591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:29:52.702777 4591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.153.20:58279/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:33971
--ranger_config_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:33971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=42277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:29:52.704766 4591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:29:52.706912 4591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:29:52.720701 4596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:52.720650 4599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:52.720942 4597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:29:52.721591 4591 server_base.cc:1034] running on GCE node
I20260929 11:29:52.722445 4591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:29:52.724161 4591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:29:52.725437 4591 hybrid_clock.cc:648] HybridClock initialized: now 1790681392725370 us; error 59 us; skew 500 ppm
I20260929 11:29:52.725873 4591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:29:52 dist-test-slave-qk05 krb5kdc[3613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681392, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:29:52.730301 4591 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:29:52.732515 4591 webserver.cc:460] Webserver started at http://127.23.153.62:42277/ using document root <none> and password file <none>
I20260929 11:29:52.733563 4591 fs_manager.cc:359] Metadata directory not provided
I20260929 11:29:52.733742 4591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 11:29:52 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681392, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
I20260929 11:29:52.796509 4591 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:29:52.800015 4607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:52.801880 4591 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260929 11:29:52.802153 4591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "ddfcabeb084a46be92ac593ebfdfeea8"
format_stamp: "Formatted at 2026-09-29 11:29:48 on dist-test-slave-qk05"
server_key: "e71063d9c22e7c015a9613d4a8ba9de3"
server_key_iv: "b203268b027a5f1b224e330b47349aa9"
server_key_version: "kuduclusterkey@0"
I20260929 11:29:52.803318 4591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:29:52.828133 4591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:29:52.834040 4591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:29:52.834379 4591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:29:52.854470 4591 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:33971
I20260929 11:29:52.854482 4658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:33971 every 8 connection(s)
I20260929 11:29:52.856144 4591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:29:52.858686 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 4591
I20260929 11:29:52.859717 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.153.20:58279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:45065
--local_ip_for_outbound_sockets=127.23.153.1
--tserver_master_addrs=127.23.153.62:33971
--webserver_port=34023
--webserver_interface=127.23.153.1
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.153.20:58279/kms with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:29:52.864295 4659 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:29:52.868639 4659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8: Bootstrap starting.
I20260929 11:29:52.897228 4659 log.cc:824] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8: Log is configured to *not* fsync() on all Append() calls
I20260929 11:29:52.908787 4659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:29:52.909552 4659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8: Bootstrap complete.
I20260929 11:29:52.920805 4659 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfcabeb084a46be92ac593ebfdfeea8" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33971 } }
I20260929 11:29:52.921819 4659 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfcabeb084a46be92ac593ebfdfeea8, State: Initialized, Role: FOLLOWER
I20260929 11:29:52.922545 4659 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfcabeb084a46be92ac593ebfdfeea8" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33971 } }
I20260929 11:29:52.922789 4659 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:29:52.922901 4659 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:29:52.923097 4659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [term 1 FOLLOWER]: Advancing to term 2
I20260929 11:29:52.927297 4659 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfcabeb084a46be92ac593ebfdfeea8" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33971 } }
I20260929 11:29:52.927860 4659 leader_election.cc:302] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddfcabeb084a46be92ac593ebfdfeea8; no voters:
I20260929 11:29:52.928490 4659 leader_election.cc:288] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 11:29:52.928642 4663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 11:29:52.929643 4663 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [term 2 LEADER]: Becoming Leader. State: Replica: ddfcabeb084a46be92ac593ebfdfeea8, State: Running, Role: LEADER
I20260929 11:29:52.930282 4663 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfcabeb084a46be92ac593ebfdfeea8" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33971 } }
I20260929 11:29:52.930672 4659 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:29:52.932855 4665 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddfcabeb084a46be92ac593ebfdfeea8. Latest consensus state: current_term: 2 leader_uuid: "ddfcabeb084a46be92ac593ebfdfeea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfcabeb084a46be92ac593ebfdfeea8" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33971 } } }
I20260929 11:29:52.933297 4665 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [sys.catalog]: This master's current role is: LEADER
I20260929 11:29:52.933840 4664 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ddfcabeb084a46be92ac593ebfdfeea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddfcabeb084a46be92ac593ebfdfeea8" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 33971 } } }
I20260929 11:29:52.934158 4664 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8 [sys.catalog]: This master's current role is: LEADER
I20260929 11:29:52.936156 4669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:29:52.943190 4669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:29:52.945405 4669 catalog_manager.cc:1259] Loaded cluster ID: ea6cccc86ccc441797ef143af695b636
I20260929 11:29:52.945688 4669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:29:52.950713 4669 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:29:52.952702 4669 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8: Loaded TSK: 0
I20260929 11:29:52.954665 4669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:29:52.958209 4659 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260929 11:29:53.383962 4661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:29:53.384459 4661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:29:53.384684 4661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:29:53.397389 4661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:29:53.397560 4661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:29:53.397754 4661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:29:53.414428 4661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.153.20:58279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:45065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=34023
--tserver_master_addrs=127.23.153.62:33971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:29:53.416671 4661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:29:53.418589 4661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:29:53.462705 4697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:53.463673 4696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:53.467334 4700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:29:53.467311 4661 server_base.cc:1034] running on GCE node
I20260929 11:29:53.468436 4661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:29:53.469908 4661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:29:53.471213 4661 hybrid_clock.cc:648] HybridClock initialized: now 1790681393471149 us; error 66 us; skew 500 ppm
I20260929 11:29:53.472343 4661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:29:53 dist-test-slave-qk05 krb5kdc[3613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681393, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:29:53.480681 4661 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:29:53.519547 4661 webserver.cc:460] Webserver started at http://127.23.153.1:34023/ using document root <none> and password file <none>
I20260929 11:29:53.520972 4661 fs_manager.cc:359] Metadata directory not provided
I20260929 11:29:53.521296 4661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 11:29:53 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681393, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
I20260929 11:29:53.579873 4661 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.002s sys 0.008s
I20260929 11:29:53.583335 4708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:53.585247 4661 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:29:53.585484 4661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "be6b7facd77445e59039bb2104b7be19"
format_stamp: "Formatted at 2026-09-29 11:29:49 on dist-test-slave-qk05"
server_key: "ca311c02bd4e83145ab239ef2d8bd470"
server_key_iv: "742a9080dfe5f3f252f9b25f4450fbd3"
server_key_version: "kuduclusterkey@0"
I20260929 11:29:53.587332 4661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:29:53.608013 4661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:29:53.613857 4661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:29:53.614187 4661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:29:53.615326 4661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:29:53.618729 4661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:29:53.618929 4661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:53.619145 4661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:29:53.619238 4661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:53.680390 4661 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:45065
I20260929 11:29:53.680428 4820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:45065 every 8 connection(s)
I20260929 11:29:53.681964 4661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:29:53.685382 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 4661
I20260929 11:29:53.686077 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.153.20:58279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:44293
--local_ip_for_outbound_sockets=127.23.153.2
--tserver_master_addrs=127.23.153.62:33971
--webserver_port=46385
--webserver_interface=127.23.153.2
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.153.20:58279/kms with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:29:53 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681393, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:29:53.758599 4825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:53.693714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:34791 (local address 127.23.153.62:33971)
0929 11:29:53.694872 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:53.694895 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:53.698479 (+ 3584us) server_negotiation.cc:407] Connection header received
0929 11:29:53.698986 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:53.699007 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:53.699632 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:53.700108 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:53.721784 (+ 21676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:53.724454 (+ 2670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:53.728126 (+ 3672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:53.729446 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:53.729674 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:53.741611 (+ 11937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:53.741624 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:53.741643 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:53.741699 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:53.745897 (+ 4198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:53.747193 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:53.747204 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:53.747212 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:53.747435 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:53.748081 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:53.748086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:53.748090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:53.748745 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:53.748949 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:53.756341 (+ 7392us) server_negotiation.cc:299] Negotiation successful
0929 11:29:53.756951 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":564,"threads_started":1}
I20260929 11:29:53.760797 4823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:53.686459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:33971 (local address 127.23.153.1:34791)
0929 11:29:53.694985 (+ 8526us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:53.695074 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0929 11:29:53.697894 (+ 2820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:29:53.719606 (+ 21712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:29:53.719626 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:29:53.720383 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:29:53.721405 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:53.721442 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:53.724810 (+ 3368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:53.724817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:53.727828 (+ 3011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:53.727859 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:53.730200 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:53.730208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:53.730525 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:53.736159 (+ 5634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:29:53.736198 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:29:53.741199 (+ 5001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:29:53.746187 (+ 4988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:53.746198 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:53.746213 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:53.746901 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:53.747604 (+ 703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:53.747609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:53.747613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:53.747914 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:53.749056 (+ 1142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:29:53.749065 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:29:53.749466 (+ 401us) client_negotiation.cc:769] Sending connection context
0929 11:29:53.749771 (+ 305us) client_negotiation.cc:240] Negotiation successful
0929 11:29:53.750705 (+ 934us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8137,"thread_start_us":12149,"threads_started":1}
I20260929 11:29:53.767478 4821 heartbeater.cc:347] Connected to a master server at 127.23.153.62:33971
I20260929 11:29:53.767889 4821 heartbeater.cc:464] Registering TS with master...
I20260929 11:29:53.768915 4821 heartbeater.cc:511] Master 127.23.153.62:33971 requested a full tablet report, sending...
W20260929 11:29:53.773249 4821 heartbeater.cc:644] Failed to heartbeat to 127.23.153.62:33971 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 11:29:53.773562 4821 heartbeater.cc:464] Registering TS with master...
I20260929 11:29:53.773917 4821 heartbeater.cc:511] Master 127.23.153.62:33971 requested a full tablet report, sending...
I20260929 11:29:53.775847 4821 heartbeater.cc:464] Registering TS with master...
I20260929 11:29:53.776307 4821 heartbeater.cc:511] Master 127.23.153.62:33971 requested a full tablet report, sending...
W20260929 11:29:53.777979 4821 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 11:29:54.166054 4824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:29:54.166528 4824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:29:54.166761 4824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:29:54.179405 4824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:29:54.179605 4824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:29:54.179804 4824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:29:54.196728 4824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.153.20:58279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:44293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=46385
--tserver_master_addrs=127.23.153.62:33971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:29:54.199527 4824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:29:54.201886 4824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:29:54.235285 4839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:54.235266 4840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:54.264878 4845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:29:54.266062 4824 server_base.cc:1034] running on GCE node
I20260929 11:29:54.267071 4824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:29:54.271142 4824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:29:54.272498 4824 hybrid_clock.cc:648] HybridClock initialized: now 1790681394272440 us; error 69 us; skew 500 ppm
I20260929 11:29:54.272869 4824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:29:54 dist-test-slave-qk05 krb5kdc[3613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681394, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:29:54.276258 4824 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:29:54.278185 4824 webserver.cc:460] Webserver started at http://127.23.153.2:46385/ using document root <none> and password file <none>
I20260929 11:29:54.278956 4824 fs_manager.cc:359] Metadata directory not provided
I20260929 11:29:54.279143 4824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 11:29:54 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681394, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
I20260929 11:29:54.326611 4824 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.008s sys 0.008s
I20260929 11:29:54.329936 4852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:54.331669 4824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 11:29:54.331919 4824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "dfb424c2a3b741bc985bf0a9be51d018"
format_stamp: "Formatted at 2026-09-29 11:29:50 on dist-test-slave-qk05"
server_key: "17c170969db065d1e3dff1f92a67ddf7"
server_key_iv: "dc3e9fc93597ec82334c2872e5914d52"
server_key_version: "kuduclusterkey@0"
I20260929 11:29:54.332996 4824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 29 11:29:54 dist-test-slave-qk05 krb5kdc[3613](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790681394, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:29:54.355158 4824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:29:54.360776 4824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:29:54.361200 4824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:29:54.363469 4824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:29:54.365175 4824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:29:54.365361 4824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 11:29:54.365522 4824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:29:54.365643 4824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:54.410039 4824 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:44293
I20260929 11:29:54.410053 4964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:44293 every 8 connection(s)
I20260929 11:29:54.412022 4824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:29:54.422549 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 4824
I20260929 11:29:54.423399 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.153.20:58279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:36957
--local_ip_for_outbound_sockets=127.23.153.3
--tserver_master_addrs=127.23.153.62:33971
--webserver_port=39623
--webserver_interface=127.23.153.3
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.153.20:58279/kms with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:29:54 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681394, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:29:54.458568 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:54.415804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:33971 (local address 127.23.153.2:37321)
0929 11:29:54.427722 (+ 11918us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:54.427795 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 11:29:54.430194 (+ 2399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:29:54.436190 (+ 5996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:29:54.436207 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:29:54.436924 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:29:54.438114 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:54.438289 (+ 175us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:54.440127 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:54.440134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:54.442282 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:54.442308 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:54.443727 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:54.443735 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:54.444053 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:54.444932 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:29:54.444967 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:29:54.448040 (+ 3073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:29:54.452765 (+ 4725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:54.452779 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:54.452804 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:54.453557 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:54.454121 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:54.454127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:54.454132 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:54.454471 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:54.455337 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:29:54.455348 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:29:54.455832 (+ 484us) client_negotiation.cc:769] Sending connection context
0929 11:29:54.456165 (+ 333us) client_negotiation.cc:240] Negotiation successful
0929 11:29:54.456781 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11520,"thread_start_us":16122,"threads_started":1}
I20260929 11:29:54.459914 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:54.414999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:37321 (local address 127.23.153.62:33971)
0929 11:29:54.435174 (+ 20175us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:54.435181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:54.435212 (+ 31us) server_negotiation.cc:407] Connection header received
0929 11:29:54.435286 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:54.435291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:54.435377 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:54.435516 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:54.438601 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:54.439879 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:54.442591 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:54.443218 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:54.443350 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:54.448496 (+ 5146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:54.448504 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:54.448510 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:54.448564 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:54.452481 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:54.453771 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:54.453779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:54.453783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:54.453956 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:54.454641 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:54.454647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:54.454651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:54.455092 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:54.455245 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:54.459225 (+ 3980us) server_negotiation.cc:299] Negotiation successful
0929 11:29:54.459521 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19943,"thread_start_us":20591,"threads_started":1}
I20260929 11:29:54.462536 4965 heartbeater.cc:347] Connected to a master server at 127.23.153.62:33971
I20260929 11:29:54.462913 4965 heartbeater.cc:464] Registering TS with master...
I20260929 11:29:54.464510 4965 heartbeater.cc:511] Master 127.23.153.62:33971 requested a full tablet report, sending...
I20260929 11:29:54.467271 4624 ts_manager.cc:194] Registered new tserver with Master: dfb424c2a3b741bc985bf0a9be51d018 (127.23.153.2:44293)
I20260929 11:29:54.469764 4624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:37321
I20260929 11:29:54.778131 4821 heartbeater.cc:464] Registering TS with master...
I20260929 11:29:54.778620 4821 heartbeater.cc:511] Master 127.23.153.62:33971 requested a full tablet report, sending...
I20260929 11:29:54.780117 4624 ts_manager.cc:194] Registered new tserver with Master: be6b7facd77445e59039bb2104b7be19 (127.23.153.1:45065)
I20260929 11:29:54.781314 4624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:34791
Sep 29 11:29:54 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681394, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
W20260929 11:29:54.951063 4970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:29:54.951642 4970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:29:54.951869 4970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:29:54.964568 4970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:29:54.964823 4970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:29:54.965040 4970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:29:54.982758 4970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.153.20:58279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:36957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=39623
--tserver_master_addrs=127.23.153.62:33971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:29:54.985066 4970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:29:54.987186 4970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:29:55.025853 4975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:55.027462 4976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:29:55.035954 4970 server_base.cc:1034] running on GCE node
W20260929 11:29:55.038197 4979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:29:55.039688 4970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:29:55.060526 4970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:29:55.064805 4970 hybrid_clock.cc:648] HybridClock initialized: now 1790681395064709 us; error 118 us; skew 500 ppm
I20260929 11:29:55.065367 4970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:29:55 dist-test-slave-qk05 krb5kdc[3613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681395, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:29:55.076748 4970 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:29:55.107481 4970 webserver.cc:460] Webserver started at http://127.23.153.3:39623/ using document root <none> and password file <none>
I20260929 11:29:55.108721 4970 fs_manager.cc:359] Metadata directory not provided
I20260929 11:29:55.108978 4970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 11:29:55 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681395, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
I20260929 11:29:55.161420 4970 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.005s sys 0.004s
I20260929 11:29:55.173282 4986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:55.175800 4970 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.006s sys 0.004s
I20260929 11:29:55.176270 4970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "8877ed09137d496c9284dd61c0d223d1"
format_stamp: "Formatted at 2026-09-29 11:29:51 on dist-test-slave-qk05"
server_key: "5a16a755266ba1cf15727ea8c1fc1450"
server_key_iv: "371bfeb5898bc18f543c9c12339c9460"
server_key_version: "kuduclusterkey@0"
I20260929 11:29:55.178783 4970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:29:55.212483 4970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:29:55.217435 4970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:29:55.217866 4970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:29:55.226042 4970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:29:55.239526 4970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:29:55.239672 4970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:55.239795 4970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:29:55.239876 4970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 11:29:55 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681394, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for HTTP/127.23.153.20@KRBTEST.COM
I20260929 11:29:55.474732 4965 heartbeater.cc:503] Master 127.23.153.62:33971 was elected leader, sending a full tablet report...
I20260929 11:29:55.512219 4970 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:36957
I20260929 11:29:55.512305 5099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:36957 every 8 connection(s)
I20260929 11:29:55.514371 4970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:29:55.514824 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 4970
I20260929 11:29:55.545120 5102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:55.516010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.23.153.62:33971)
0929 11:29:55.520775 (+ 4765us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:55.520781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:55.520803 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:29:55.520878 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:55.520883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:55.521022 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:55.521162 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:55.524900 (+ 3738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.526506 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.529348 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.530136 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.530293 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:55.535049 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:55.535057 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:55.535062 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:55.535126 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:55.541004 (+ 5878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:55.542258 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:55.542272 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:55.542276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:55.542470 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:55.543005 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:55.543035 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:55.543039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:55.543478 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:55.543798 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:55.544426 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 11:29:55.544721 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4615,"thread_start_us":5328,"threads_started":1}
Sep 29 11:29:55 dist-test-slave-qk05 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681395, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:29:55.568046 5104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:55.519626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:56043 (local address 127.23.153.62:33971)
0929 11:29:55.543742 (+ 24116us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:55.543749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:55.543774 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:29:55.548526 (+ 4752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:55.548532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:55.548664 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:55.548771 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:55.550939 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.551841 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.553726 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.554190 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.554316 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:55.558425 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:55.558431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:55.558435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:55.558476 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:55.562193 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:55.563483 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:55.563490 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:55.563494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:55.563663 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:55.564757 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:55.564765 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:55.564770 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:55.565304 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:55.565498 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:55.566932 (+ 1434us) server_negotiation.cc:299] Negotiation successful
0929 11:29:55.567462 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23927,"thread_start_us":24126,"threads_started":1}
I20260929 11:29:55.569036 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:55.520898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:33971 (local address 127.23.153.3:56043)
0929 11:29:55.540937 (+ 20039us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:55.541019 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0929 11:29:55.545029 (+ 4010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:29:55.549396 (+ 4367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:29:55.549411 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:29:55.549971 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:29:55.550651 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:55.550686 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.552049 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.552054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:55.553528 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:55.553546 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.554479 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.554488 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:55.554749 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:55.555503 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:29:55.555534 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:29:55.557995 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:29:55.562478 (+ 4483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:55.562493 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:55.562522 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:55.563350 (+ 828us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:55.564025 (+ 675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:55.564033 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:55.564038 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:55.564525 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:55.565593 (+ 1068us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:29:55.565603 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:29:55.566237 (+ 634us) client_negotiation.cc:769] Sending connection context
0929 11:29:55.566624 (+ 387us) client_negotiation.cc:240] Negotiation successful
0929 11:29:55.567370 (+ 746us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19541,"thread_start_us":21987,"threads_started":1}
I20260929 11:29:55.571224 5100 heartbeater.cc:347] Connected to a master server at 127.23.153.62:33971
I20260929 11:29:55.571602 5100 heartbeater.cc:464] Registering TS with master...
I20260929 11:29:55.572356 5100 heartbeater.cc:511] Master 127.23.153.62:33971 requested a full tablet report, sending...
I20260929 11:29:55.574792 4624 ts_manager.cc:194] Registered new tserver with Master: 8877ed09137d496c9284dd61c0d223d1 (127.23.153.3:36957)
I20260929 11:29:55.576076 4624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:56043
I20260929 11:29:55.578313 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:29:55.604094 5104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:55.588024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34354 (local address 127.23.153.62:33971)
0929 11:29:55.588218 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:55.588223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:55.588248 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:29:55.588539 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:55.588546 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:55.588632 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:55.588694 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:55.590413 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.591217 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.592858 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.593294 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.593414 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:55.596642 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:55.596648 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:55.596652 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:55.596700 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:55.601047 (+ 4347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:55.601931 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:55.601937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:55.601940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:55.602106 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:55.602510 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:55.602515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:55.602517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:55.602835 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:55.603105 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:55.603489 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 11:29:55.603759 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:29:55.611591 4624 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:29:55.621398 4624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:29:55.680152 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:55.640581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.23.153.2:44293)
0929 11:29:55.671625 (+ 31044us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:55.671632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:55.671657 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:29:55.671734 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:55.671740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:55.671911 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:55.672100 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:29:55.672852 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.674354 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.676711 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.678770 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.678930 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:55.679248 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:55.679590 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 11:29:55.679794 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30880,"thread_start_us":30702,"threads_started":1}
I20260929 11:29:55.680572 5122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:55.640333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:44293 (local address 127.0.0.1:52252)
0929 11:29:55.653818 (+ 13485us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:55.653865 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 11:29:55.654115 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:29:55.672142 (+ 18027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:29:55.672147 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:29:55.672179 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:29:55.672634 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:55.672647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.674527 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.674533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:55.676494 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:55.676511 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.679092 (+ 2581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.679101 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:55.679340 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:55.679384 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 11:29:55.679518 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 11:29:55.680196 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13151,"mutex_wait_us":66,"thread_start_us":12960,"threads_started":1}
I20260929 11:29:55.681414 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:55.641127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52952 (local address 127.23.153.1:45065)
0929 11:29:55.669011 (+ 27884us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:55.669021 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:55.669671 (+ 650us) server_negotiation.cc:407] Connection header received
0929 11:29:55.669906 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:55.669916 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:55.670230 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:55.670463 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:29:55.671613 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.673188 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.676472 (+ 3284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.678261 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.678428 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:55.680555 (+ 2127us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:55.680748 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 11:29:55.680946 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27688,"thread_start_us":28678,"threads_started":1}
I20260929 11:29:55.681953 5125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:55.640593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:45065 (local address 127.0.0.1:52952)
0929 11:29:55.669508 (+ 28915us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:55.669530 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:29:55.669717 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:29:55.670801 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:29:55.670809 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:29:55.670840 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:29:55.671342 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:55.671362 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.673366 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.673372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:55.676226 (+ 2854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:55.676248 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.679648 (+ 3400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.679661 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:55.679842 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:55.679877 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:29:55.679954 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 11:29:55.680123 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28786,"mutex_wait_us":49,"thread_start_us":29583,"threads_started":1}
I20260929 11:29:55.684573 4900 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6e713615df4bd8b64803da7fdff2d5 (DEFAULT_TABLE table=test-table [id=61ba99d617c648b983cb833262405eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:29:55.686379 4900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b6e713615df4bd8b64803da7fdff2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:29:55.687495 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:55.641536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.23.153.3:36957)
0929 11:29:55.668248 (+ 26712us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:55.668262 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:55.670579 (+ 2317us) server_negotiation.cc:407] Connection header received
0929 11:29:55.670889 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:55.670898 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:55.671375 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:55.671556 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:29:55.672627 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.673889 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.677229 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.685426 (+ 8197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.685667 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:55.685998 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:55.686796 (+ 798us) server_negotiation.cc:299] Negotiation successful
0929 11:29:55.687090 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26434,"thread_start_us":26748,"threads_started":1}
I20260929 11:29:55.688197 5124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:55.640842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:36957 (local address 127.0.0.1:32920)
0929 11:29:55.670455 (+ 29613us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:55.670475 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:29:55.670695 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:29:55.671944 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:29:55.671949 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:29:55.671976 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:29:55.672409 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:55.672423 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.674493 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.674498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:55.676952 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:55.676968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.685889 (+ 8921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.685898 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:55.686161 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:55.686204 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 11:29:55.686325 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 11:29:55.686534 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29476,"mutex_wait_us":51,"thread_start_us":28391,"threads_started":1}
I20260929 11:29:55.687969 4756 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6e713615df4bd8b64803da7fdff2d5 (DEFAULT_TABLE table=test-table [id=61ba99d617c648b983cb833262405eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:29:55.689640 4756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b6e713615df4bd8b64803da7fdff2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:29:55.691821 5035 tablet_service.cc:1467] Processing CreateTablet for tablet 3b6e713615df4bd8b64803da7fdff2d5 (DEFAULT_TABLE table=test-table [id=61ba99d617c648b983cb833262405eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:29:55.693454 5035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b6e713615df4bd8b64803da7fdff2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:29:55.709106 5128 tablet_bootstrap.cc:492] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018: Bootstrap starting.
I20260929 11:29:55.711760 5129 tablet_bootstrap.cc:492] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1: Bootstrap starting.
I20260929 11:29:55.714226 5128 tablet_bootstrap.cc:654] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018: Neither blocks nor log segments found. Creating new log.
I20260929 11:29:55.714725 5129 tablet_bootstrap.cc:654] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1: Neither blocks nor log segments found. Creating new log.
I20260929 11:29:55.715502 5130 tablet_bootstrap.cc:492] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19: Bootstrap starting.
I20260929 11:29:55.716058 5129 log.cc:824] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1: Log is configured to *not* fsync() on all Append() calls
I20260929 11:29:55.716447 5128 log.cc:824] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018: Log is configured to *not* fsync() on all Append() calls
I20260929 11:29:55.719691 5130 tablet_bootstrap.cc:654] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19: Neither blocks nor log segments found. Creating new log.
I20260929 11:29:55.720382 5128 tablet_bootstrap.cc:492] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018: No bootstrap required, opened a new log
I20260929 11:29:55.720423 5129 tablet_bootstrap.cc:492] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1: No bootstrap required, opened a new log
I20260929 11:29:55.720670 5128 ts_tablet_manager.cc:1397] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260929 11:29:55.720782 5129 ts_tablet_manager.cc:1397] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260929 11:29:55.721200 5130 log.cc:824] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19: Log is configured to *not* fsync() on all Append() calls
I20260929 11:29:55.724836 5130 tablet_bootstrap.cc:492] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19: No bootstrap required, opened a new log
I20260929 11:29:55.725180 5130 ts_tablet_manager.cc:1397] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260929 11:29:55.729111 5128 raft_consensus.cc:348] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb424c2a3b741bc985bf0a9be51d018" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44293 } } peers { permanent_uuid: "be6b7facd77445e59039bb2104b7be19" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45065 } } peers { permanent_uuid: "8877ed09137d496c9284dd61c0d223d1" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36957 } }
I20260929 11:29:55.729548 5128 raft_consensus.cc:374] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:29:55.729657 5128 raft_consensus.cc:729] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb424c2a3b741bc985bf0a9be51d018, State: Initialized, Role: FOLLOWER
I20260929 11:29:55.730289 5128 consensus_queue.cc:260] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb424c2a3b741bc985bf0a9be51d018" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44293 } } peers { permanent_uuid: "be6b7facd77445e59039bb2104b7be19" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45065 } } peers { permanent_uuid: "8877ed09137d496c9284dd61c0d223d1" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36957 } }
I20260929 11:29:55.731251 5129 raft_consensus.cc:348] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb424c2a3b741bc985bf0a9be51d018" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44293 } } peers { permanent_uuid: "be6b7facd77445e59039bb2104b7be19" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45065 } } peers { permanent_uuid: "8877ed09137d496c9284dd61c0d223d1" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36957 } }
I20260929 11:29:55.731717 5129 raft_consensus.cc:374] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:29:55.731804 5129 raft_consensus.cc:729] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8877ed09137d496c9284dd61c0d223d1, State: Initialized, Role: FOLLOWER
I20260929 11:29:55.732462 5128 ts_tablet_manager.cc:1428] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260929 11:29:55.732532 5129 consensus_queue.cc:260] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb424c2a3b741bc985bf0a9be51d018" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44293 } } peers { permanent_uuid: "be6b7facd77445e59039bb2104b7be19" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45065 } } peers { permanent_uuid: "8877ed09137d496c9284dd61c0d223d1" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36957 } }
I20260929 11:29:55.734051 5100 heartbeater.cc:503] Master 127.23.153.62:33971 was elected leader, sending a full tablet report...
I20260929 11:29:55.734716 5129 ts_tablet_manager.cc:1428] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260929 11:29:55.736485 5130 raft_consensus.cc:348] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb424c2a3b741bc985bf0a9be51d018" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44293 } } peers { permanent_uuid: "be6b7facd77445e59039bb2104b7be19" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45065 } } peers { permanent_uuid: "8877ed09137d496c9284dd61c0d223d1" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36957 } }
I20260929 11:29:55.736986 5130 raft_consensus.cc:374] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:29:55.737085 5130 raft_consensus.cc:729] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be6b7facd77445e59039bb2104b7be19, State: Initialized, Role: FOLLOWER
I20260929 11:29:55.737921 5130 consensus_queue.cc:260] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb424c2a3b741bc985bf0a9be51d018" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44293 } } peers { permanent_uuid: "be6b7facd77445e59039bb2104b7be19" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45065 } } peers { permanent_uuid: "8877ed09137d496c9284dd61c0d223d1" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36957 } }
I20260929 11:29:55.739502 4821 heartbeater.cc:503] Master 127.23.153.62:33971 was elected leader, sending a full tablet report...
I20260929 11:29:55.740386 5130 ts_tablet_manager.cc:1428] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260929 11:29:55.784013 5134 raft_consensus.cc:482] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:29:55.784403 5134 raft_consensus.cc:504] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb424c2a3b741bc985bf0a9be51d018" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44293 } } peers { permanent_uuid: "be6b7facd77445e59039bb2104b7be19" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45065 } } peers { permanent_uuid: "8877ed09137d496c9284dd61c0d223d1" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36957 } }
I20260929 11:29:55.786270 5134 leader_election.cc:288] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be6b7facd77445e59039bb2104b7be19 (127.23.153.1:45065), 8877ed09137d496c9284dd61c0d223d1 (127.23.153.3:36957)
I20260929 11:29:55.793256 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:55.786869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:47981 (local address 127.23.153.1:45065)
0929 11:29:55.787092 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:55.787099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:55.787669 (+ 570us) server_negotiation.cc:407] Connection header received
0929 11:29:55.787856 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:55.787860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:55.787918 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:55.788002 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:29:55.788755 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.789512 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.791485 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.792324 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.792484 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:55.792564 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:55.792867 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 11:29:55.793011 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 11:29:55.793556 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:55.786854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:34183 (local address 127.23.153.3:36957)
0929 11:29:55.787070 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:55.787077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:55.787719 (+ 642us) server_negotiation.cc:407] Connection header received
0929 11:29:55.787907 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:55.787911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:55.787973 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:55.788054 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:29:55.788724 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.789708 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.791791 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.792622 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.792784 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:55.792861 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:55.793147 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 11:29:55.793291 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 11:29:55.793648 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:55.786532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:45065 (local address 127.23.153.2:47981)
0929 11:29:55.787543 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:55.787565 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:29:55.787698 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:29:55.788030 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:29:55.788132 (+ 102us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:29:55.788157 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:29:55.788614 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:55.788624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.789662 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.789666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:55.791301 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:55.791322 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.792552 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.792558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:55.792698 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:55.792724 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 11:29:55.792812 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 11:29:55.792963 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":24,"thread_start_us":633,"threads_started":1}
I20260929 11:29:55.793702 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:55.786767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:36957 (local address 127.23.153.2:34183)
0929 11:29:55.787548 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:55.787562 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 11:29:55.787773 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:29:55.788092 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:29:55.788096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:29:55.788120 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:29:55.788534 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:55.788548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.789974 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.789979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:55.791598 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:55.791612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.792849 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.792855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:55.792994 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:55.793018 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 11:29:55.793098 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 11:29:55.793310 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":40,"spinlock_wait_cycles":25984,"thread_start_us":524,"threads_started":1}
I20260929 11:29:55.794853 5055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6e713615df4bd8b64803da7fdff2d5" candidate_uuid: "dfb424c2a3b741bc985bf0a9be51d018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8877ed09137d496c9284dd61c0d223d1" is_pre_election: true
I20260929 11:29:55.794826 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6e713615df4bd8b64803da7fdff2d5" candidate_uuid: "dfb424c2a3b741bc985bf0a9be51d018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be6b7facd77445e59039bb2104b7be19" is_pre_election: true
I20260929 11:29:55.795527 4776 raft_consensus.cc:2454] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfb424c2a3b741bc985bf0a9be51d018 in term 0.
I20260929 11:29:55.795526 5055 raft_consensus.cc:2454] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfb424c2a3b741bc985bf0a9be51d018 in term 0.
I20260929 11:29:55.796336 4856 leader_election.cc:302] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8877ed09137d496c9284dd61c0d223d1, be6b7facd77445e59039bb2104b7be19, dfb424c2a3b741bc985bf0a9be51d018; no voters:
I20260929 11:29:55.796746 5134 raft_consensus.cc:2781] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:29:55.796888 5134 raft_consensus.cc:482] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:29:55.796985 5134 raft_consensus.cc:3037] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:29:55.799335 5134 raft_consensus.cc:504] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb424c2a3b741bc985bf0a9be51d018" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44293 } } peers { permanent_uuid: "be6b7facd77445e59039bb2104b7be19" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45065 } } peers { permanent_uuid: "8877ed09137d496c9284dd61c0d223d1" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36957 } }
W20260929 11:29:55.799860 5103 tablet.cc:2369] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:29:55.800089 5134 leader_election.cc:288] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [CANDIDATE]: Term 1 election: Requested vote from peers be6b7facd77445e59039bb2104b7be19 (127.23.153.1:45065), 8877ed09137d496c9284dd61c0d223d1 (127.23.153.3:36957)
I20260929 11:29:55.800746 4776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6e713615df4bd8b64803da7fdff2d5" candidate_uuid: "dfb424c2a3b741bc985bf0a9be51d018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be6b7facd77445e59039bb2104b7be19"
I20260929 11:29:55.800750 5055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b6e713615df4bd8b64803da7fdff2d5" candidate_uuid: "dfb424c2a3b741bc985bf0a9be51d018" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8877ed09137d496c9284dd61c0d223d1"
I20260929 11:29:55.800962 4776 raft_consensus.cc:3037] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:29:55.800959 5055 raft_consensus.cc:3037] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:29:55.803514 5055 raft_consensus.cc:2454] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb424c2a3b741bc985bf0a9be51d018 in term 1.
I20260929 11:29:55.803539 4776 raft_consensus.cc:2454] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb424c2a3b741bc985bf0a9be51d018 in term 1.
I20260929 11:29:55.804078 4854 leader_election.cc:302] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8877ed09137d496c9284dd61c0d223d1, be6b7facd77445e59039bb2104b7be19, dfb424c2a3b741bc985bf0a9be51d018; no voters:
I20260929 11:29:55.804467 5134 raft_consensus.cc:2781] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:29:55.804912 5134 raft_consensus.cc:686] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [term 1 LEADER]: Becoming Leader. State: Replica: dfb424c2a3b741bc985bf0a9be51d018, State: Running, Role: LEADER
I20260929 11:29:55.805456 5134 consensus_queue.cc:237] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb424c2a3b741bc985bf0a9be51d018" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44293 } } peers { permanent_uuid: "be6b7facd77445e59039bb2104b7be19" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45065 } } peers { permanent_uuid: "8877ed09137d496c9284dd61c0d223d1" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36957 } }
I20260929 11:29:55.810384 4624 catalog_manager.cc:5495] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfb424c2a3b741bc985bf0a9be51d018 (127.23.153.2). New cstate: current_term: 1 leader_uuid: "dfb424c2a3b741bc985bf0a9be51d018" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfb424c2a3b741bc985bf0a9be51d018" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 44293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be6b7facd77445e59039bb2104b7be19" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8877ed09137d496c9284dd61c0d223d1" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36957 } health_report { overall_health: UNKNOWN } } }
I20260929 11:29:55.847633 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:55.841647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52260 (local address 127.23.153.2:44293)
0929 11:29:55.841832 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:55.841836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:55.841856 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:29:55.841964 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:55.841967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:55.842053 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:55.842212 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:29:55.842594 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.843377 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.844873 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.845311 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.845431 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:55.846311 (+ 880us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:29:55.846861 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:29:55.846988 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:55.847251 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 11:29:55.847340 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:29:55.857218 5055 raft_consensus.cc:1261] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1 [term 1 FOLLOWER]: Refusing update from remote peer dfb424c2a3b741bc985bf0a9be51d018: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:29:55.857296 4776 raft_consensus.cc:1261] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19 [term 1 FOLLOWER]: Refusing update from remote peer dfb424c2a3b741bc985bf0a9be51d018: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:29:55.858413 5134 consensus_queue.cc:1035] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be6b7facd77445e59039bb2104b7be19" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 45065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:29:55.858752 5139 consensus_queue.cc:1035] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8877ed09137d496c9284dd61c0d223d1" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 36957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:29:55.875800 5142 mvcc.cc:205] Tried to move back new op lower bound from 7334630997419687936 to 7334630997223190528. Current Snapshot: MvccSnapshot[applied={T|T < 7334630997419687936}]
I20260929 11:29:55.882133 5143 mvcc.cc:205] Tried to move back new op lower bound from 7334630997419687936 to 7334630997223190528. Current Snapshot: MvccSnapshot[applied={T|T < 7334630997419687936}]
I20260929 11:29:55.897861 4623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:29:55.898321 4623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:29:55.902478 4623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ddfcabeb084a46be92ac593ebfdfeea8: Sending DeleteTablet for 3 replicas of tablet 3b6e713615df4bd8b64803da7fdff2d5
I20260929 11:29:55.903885 5035 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b6e713615df4bd8b64803da7fdff2d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:29:55 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:32920
I20260929 11:29:55.904071 4900 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b6e713615df4bd8b64803da7fdff2d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:29:55 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:52252
I20260929 11:29:55.904983 5152 tablet_replica.cc:330] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1: stopping tablet replica
I20260929 11:29:55.905088 4756 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b6e713615df4bd8b64803da7fdff2d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:29:55 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:52952
I20260929 11:29:55.905228 5153 tablet_replica.cc:330] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018: stopping tablet replica
I20260929 11:29:55.905638 5153 raft_consensus.cc:2229] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:29:55.905633 5152 raft_consensus.cc:2229] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:29:55.906267 5152 raft_consensus.cc:2258] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:29:55.906342 5153 raft_consensus.cc:2258] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:29:55.907148 5154 tablet_replica.cc:330] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19: stopping tablet replica
I20260929 11:29:55.907642 5154 raft_consensus.cc:2229] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:29:55.908196 5154 raft_consensus.cc:2258] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:29:55.909798 5153 ts_tablet_manager.cc:1905] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:29:55.910305 5154 ts_tablet_manager.cc:1905] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:29:55.913654 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 4661
I20260929 11:29:55.916024 5152 ts_tablet_manager.cc:1905] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:29:55.917793 5154 ts_tablet_manager.cc:1918] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:29:55.917994 5154 log.cc:1196] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/3b6e713615df4bd8b64803da7fdff2d5
I20260929 11:29:55.918530 5154 ts_tablet_manager.cc:1939] T 3b6e713615df4bd8b64803da7fdff2d5 P be6b7facd77445e59039bb2104b7be19: Deleting consensus metadata
I20260929 11:29:55.920779 4609 catalog_manager.cc:4841] TS be6b7facd77445e59039bb2104b7be19 (127.23.153.1:45065): tablet 3b6e713615df4bd8b64803da7fdff2d5 (table test-table [id=61ba99d617c648b983cb833262405eb2]) successfully deleted
I20260929 11:29:55.926275 5152 ts_tablet_manager.cc:1918] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:29:55.926430 5153 ts_tablet_manager.cc:1918] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:29:55.926661 5153 log.cc:1196] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/3b6e713615df4bd8b64803da7fdff2d5
I20260929 11:29:55.926770 5152 log.cc:1196] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/3b6e713615df4bd8b64803da7fdff2d5
I20260929 11:29:55.927364 5153 ts_tablet_manager.cc:1939] T 3b6e713615df4bd8b64803da7fdff2d5 P dfb424c2a3b741bc985bf0a9be51d018: Deleting consensus metadata
I20260929 11:29:55.927626 5152 ts_tablet_manager.cc:1939] T 3b6e713615df4bd8b64803da7fdff2d5 P 8877ed09137d496c9284dd61c0d223d1: Deleting consensus metadata
I20260929 11:29:55.929869 4611 catalog_manager.cc:4841] TS 8877ed09137d496c9284dd61c0d223d1 (127.23.153.3:36957): tablet 3b6e713615df4bd8b64803da7fdff2d5 (table test-table [id=61ba99d617c648b983cb833262405eb2]) successfully deleted
I20260929 11:29:55.930624 4610 catalog_manager.cc:4841] TS dfb424c2a3b741bc985bf0a9be51d018 (127.23.153.2:44293): tablet 3b6e713615df4bd8b64803da7fdff2d5 (table test-table [id=61ba99d617c648b983cb833262405eb2]) successfully deleted
I20260929 11:29:55.935191 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:55.916373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.23.153.1:45065)
0929 11:29:55.916561 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:55.916567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:55.916590 (+ 23us) server_negotiation.cc:407] Connection header received
0929 11:29:55.916653 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:55.916658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:55.916756 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:55.916862 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:55.920779 (+ 3917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.922009 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.923913 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:55.924578 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:55.924738 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:55.928364 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:55.928386 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:55.928408 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:55.928573 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:55.932217 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:55.933034 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:55.933041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:55.933044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:55.933191 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:55.933568 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:55.933572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:55.933574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:55.933942 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:55.934242 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:55.934581 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 11:29:55.934814 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:29:55.935899 4816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:29:56.034143 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 4661
I20260929 11:29:56.048964 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 4824
I20260929 11:29:56.065490 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:56.049788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52268 (local address 127.23.153.2:44293)
0929 11:29:56.049965 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:56.049974 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:56.050027 (+ 53us) server_negotiation.cc:407] Connection header received
0929 11:29:56.050234 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:56.050238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:56.050302 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:56.050391 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:56.051792 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:56.052667 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:56.054016 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:56.054542 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:56.054679 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:56.058475 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:56.058493 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:56.058511 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:56.058561 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:56.062240 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:56.063308 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:56.063316 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:56.063318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:56.063534 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:56.063949 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:56.063952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:56.063954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:56.064365 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:56.064583 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:56.064967 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 11:29:56.065171 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 11:29:56.066313 4960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:29:56.183290 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 4824
I20260929 11:29:56.201157 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 4970
I20260929 11:29:56.216410 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:56.202527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.23.153.3:36957)
0929 11:29:56.202705 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:56.202710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:56.202729 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:29:56.202849 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:56.202854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:56.202917 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:56.202991 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:56.204454 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:56.205277 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:56.206756 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:56.207444 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:56.207559 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:56.210259 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:56.210276 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:56.210294 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:56.210334 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:56.213416 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:56.214250 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:56.214259 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:56.214261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:56.214410 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:56.214847 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:56.214852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:56.214854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:56.215390 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:56.215681 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:56.215924 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 11:29:56.216126 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:29:56.217087 5095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:29:56.310379 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 4970
I20260929 11:29:56.327737 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 4591
I20260929 11:29:56.329136 4654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:29:56.446884 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 4591
I20260929 11:29:56.468031 24164 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 11:29:56.804926 24164 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 11:29:56.805146 24164 mini_ranger.cc:66] Stopping Ranger...
I20260929 11:29:57.142122 24164 mini_ranger.cc:68] Stopped Ranger
2026-09-29 11:29:57.142 UTC [3643] LOG: received smart shutdown request
2026-09-29 11:29:57.146 UTC [3643] LOG: background worker "logical replication launcher" (PID 3652) exited with exit code 1
2026-09-29 11:29:57.146 UTC [3647] LOG: shutting down
2026-09-29 11:29:57.301 UTC [3643] LOG: database system is shut down
2026-09-29T11:29:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (59338 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:29:57 dist-test-slave-qk05 krb5kdc[5167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:29:57 dist-test-slave-qk05 krb5kdc[5167](info): set up 2 sockets
Sep 29 11:29:57 dist-test-slave-qk05 krb5kdc[5167](info): commencing operation
krb5kdc: starting...
W20260929 11:29:58.455559 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:29:58 dist-test-slave-qk05 krb5kdc[5167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:29:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:29:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:29:58.698889 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:34561
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:34561,127.23.153.61:35389,127.23.153.60:37725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:29:59.062454 5180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:29:59.062799 5180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:29:59.062937 5180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:29:59.070711 5180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:29:59.070863 5180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:29:59.070940 5180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:29:59.071046 5180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:29:59.071153 5180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:29:59.081754 5180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:34561,127.23.153.61:35389,127.23.153.60:37725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:34561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:29:59.083273 5180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:29:59.084650 5180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:29:59.097626 5186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:59.097584 5188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:59.097661 5185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:29:59.098454 5180 server_base.cc:1034] running on GCE node
I20260929 11:29:59.099327 5180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:29:59.100850 5180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:29:59.102113 5180 hybrid_clock.cc:648] HybridClock initialized: now 1790681399102036 us; error 45 us; skew 500 ppm
I20260929 11:29:59.102527 5180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:29:59 dist-test-slave-qk05 krb5kdc[5167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681399, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:29:59.106088 5180 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:29:59.108217 5180 webserver.cc:460] Webserver started at http://127.23.153.62:34183/ using document root <none> and password file <none>
I20260929 11:29:59.109076 5180 fs_manager.cc:359] Metadata directory not provided
I20260929 11:29:59.109215 5180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:29:59.109508 5180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:29:59.112847 5180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "141b13b505fe40c497a3d133d0067aae"
format_stamp: "Formatted at 2026-09-29 11:29:59 on dist-test-slave-qk05"
server_key: "2b5d81b39622caa0291c6e869e14f9a3"
server_key_iv: "5f7cd82f97d23920a7fe8132c9c03fdd"
server_key_version: "encryptionkey@0"
I20260929 11:29:59.113722 5180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "141b13b505fe40c497a3d133d0067aae"
format_stamp: "Formatted at 2026-09-29 11:29:59 on dist-test-slave-qk05"
server_key: "2b5d81b39622caa0291c6e869e14f9a3"
server_key_iv: "5f7cd82f97d23920a7fe8132c9c03fdd"
server_key_version: "encryptionkey@0"
I20260929 11:29:59.119987 5180 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 11:29:59.124516 5195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:59.126147 5180 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:29:59.126473 5180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "141b13b505fe40c497a3d133d0067aae"
format_stamp: "Formatted at 2026-09-29 11:29:59 on dist-test-slave-qk05"
server_key: "2b5d81b39622caa0291c6e869e14f9a3"
server_key_iv: "5f7cd82f97d23920a7fe8132c9c03fdd"
server_key_version: "encryptionkey@0"
I20260929 11:29:59.126758 5180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:29:59.140348 5180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:29:59.145026 5180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:29:59.145352 5180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:29:59.166798 5180 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:34561
I20260929 11:29:59.166821 5246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:34561 every 8 connection(s)
I20260929 11:29:59.168391 5180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:29:59.172669 5247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:29:59.174183 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 5180
I20260929 11:29:59.174437 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:29:59.175264 24164 external_mini_cluster.cc:1428] Setting key 0177ab99bc08e08a033644acb43ed389
I20260929 11:29:59.182456 5247 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.153.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.61@KRBTEST.COM" created.
Sep 29 11:29:59 dist-test-slave-qk05 krb5kdc[5167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681399, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:29:59.212210 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.186568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:34561 (local address 127.0.0.1:55618)
0929 11:29:59.187551 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.187607 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 11:29:59.189951 (+ 2344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:29:59.191152 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:29:59.191158 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:29:59.191801 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:29:59.192173 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:59.192194 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:59.194553 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:59.194559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:59.195848 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:59.195876 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:59.197694 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:59.197703 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:59.197920 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:59.198809 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:29:59.198841 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:29:59.202387 (+ 3546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:29:59.206904 (+ 4517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:59.206920 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:59.206932 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:59.207531 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:59.208103 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:59.208107 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:59.208109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:59.208304 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:59.209329 (+ 1025us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:29:59.209338 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:29:59.209613 (+ 275us) client_negotiation.cc:769] Sending connection context
0929 11:29:59.209939 (+ 326us) client_negotiation.cc:240] Negotiation successful
0929 11:29:59.210694 (+ 755us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":226,"thread_start_us":426,"threads_started":1}
I20260929 11:29:59.212230 5251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.188178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55618 (local address 127.23.153.62:34561)
0929 11:29:59.188954 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:59.188967 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:59.189027 (+ 60us) server_negotiation.cc:407] Connection header received
0929 11:29:59.190460 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:59.190469 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:59.190820 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:59.191100 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:59.192433 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:59.194304 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:59.196057 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:59.197157 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:59.197398 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:59.202747 (+ 5349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:59.202760 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:59.202779 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:59.202833 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:59.206666 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:59.207719 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:59.207730 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:59.207741 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:59.207974 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:59.208418 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:59.208423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:59.208426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:59.209168 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:59.209362 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:59.210071 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 11:29:59.210548 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":549,"thread_start_us":365,"threads_started":1}
I20260929 11:29:59.215739 5247 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:29:59.217123 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.216643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35389 (local address 127.0.0.1:54712)
0929 11:29:59.216848 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.216980 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260929 11:29:59.217613 5199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35389: connect: Connection refused (error 111)
I20260929 11:29:59.219717 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.219152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35389 (local address 127.0.0.1:54714)
0929 11:29:59.219444 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.219534 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 11:29:59.220052 5247 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.61:35389: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35389: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.61.keytab.
Entry for principal kudu/127.23.153.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.61.keytab.
I20260929 11:29:59.236119 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.61:35389
--webserver_interface=127.23.153.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.61.keytab
--principal=kudu/127.23.153.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:34561,127.23.153.61:35389,127.23.153.60:37725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:29:59.269454 5247 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } attempt: 1
I20260929 11:29:59.270713 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.270229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35389 (local address 127.0.0.1:54716)
0929 11:29:59.270447 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.270575 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 11:29:59.271790 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.271402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35389 (local address 127.0.0.1:54724)
0929 11:29:59.271559 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.271639 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 11:29:59.272001 5247 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.61:35389: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35389: connect: Connection refused (error 111)
I20260929 11:29:59.340324 5247 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } attempt: 2
I20260929 11:29:59.341666 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.341110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35389 (local address 127.0.0.1:54736)
0929 11:29:59.341415 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.341515 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 11:29:59.342734 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.342348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35389 (local address 127.0.0.1:54738)
0929 11:29:59.342502 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.342593 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 11:29:59.342942 5247 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.61:35389: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35389: connect: Connection refused (error 111)
I20260929 11:29:59.434327 5247 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } attempt: 3
I20260929 11:29:59.435736 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.435208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35389 (local address 127.0.0.1:54742)
0929 11:29:59.435489 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.435597 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260929 11:29:59.436906 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.436502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35389 (local address 127.0.0.1:54756)
0929 11:29:59.436662 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.436743 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 11:29:59.437139 5247 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.61:35389: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35389: connect: Connection refused (error 111)
I20260929 11:29:59.580521 5247 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } attempt: 4
I20260929 11:29:59.581962 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.581273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35389 (local address 127.0.0.1:54766)
0929 11:29:59.581713 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.581815 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273}
I20260929 11:29:59.583226 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.582782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35389 (local address 127.0.0.1:54772)
0929 11:29:59.582929 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.583075 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 11:29:59.583472 5247 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.61:35389: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35389: connect: Connection refused (error 111)
W20260929 11:29:59.613193 5254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:29:59.613534 5254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:29:59.613677 5254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:29:59.622603 5254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:29:59.622774 5254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:29:59.622836 5254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:29:59.622922 5254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:29:59.622984 5254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:29:59.634775 5254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:34561,127.23.153.61:35389,127.23.153.60:37725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.61.keytab
--principal=kudu/127.23.153.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.61:35389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.153.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:29:59.636309 5254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:29:59.637938 5254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:29:59.653622 5259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:59.653683 5262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:29:59.653646 5260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:29:59.654932 5254 server_base.cc:1034] running on GCE node
I20260929 11:29:59.655978 5254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:29:59.657902 5254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:29:59.659348 5254 hybrid_clock.cc:648] HybridClock initialized: now 1790681399659261 us; error 49 us; skew 500 ppm
I20260929 11:29:59.659771 5254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:29:59 dist-test-slave-qk05 krb5kdc[5167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681399, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:29:59.663152 5254 init.cc:377] Logged in from keytab as kudu/127.23.153.61@KRBTEST.COM (short username kudu)
I20260929 11:29:59.664849 5254 webserver.cc:460] Webserver started at http://127.23.153.61:38905/ using document root <none> and password file <none>
I20260929 11:29:59.665745 5254 fs_manager.cc:359] Metadata directory not provided
I20260929 11:29:59.665889 5254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:29:59.666162 5254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:29:59.670037 5254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-1/data/instance:
uuid: "9552f26c9c9a40deadcbd25184ee4e03"
format_stamp: "Formatted at 2026-09-29 11:29:59 on dist-test-slave-qk05"
server_key: "99cccde76f793e2a1db3fcdc82b8d035"
server_key_iv: "2ad9cb95376f6771fc7e0a88c0b13c48"
server_key_version: "encryptionkey@0"
I20260929 11:29:59.670934 5254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-1/wal/instance:
uuid: "9552f26c9c9a40deadcbd25184ee4e03"
format_stamp: "Formatted at 2026-09-29 11:29:59 on dist-test-slave-qk05"
server_key: "99cccde76f793e2a1db3fcdc82b8d035"
server_key_iv: "2ad9cb95376f6771fc7e0a88c0b13c48"
server_key_version: "encryptionkey@0"
I20260929 11:29:59.677570 5254 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:29:59.682358 5269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:29:59.684505 5254 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:29:59.684816 5254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-1/wal
uuid: "9552f26c9c9a40deadcbd25184ee4e03"
format_stamp: "Formatted at 2026-09-29 11:29:59 on dist-test-slave-qk05"
server_key: "99cccde76f793e2a1db3fcdc82b8d035"
server_key_iv: "2ad9cb95376f6771fc7e0a88c0b13c48"
server_key_version: "encryptionkey@0"
I20260929 11:29:59.685039 5254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:29:59.721801 5254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:29:59.726416 5254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:29:59.726701 5254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:29:59.746392 5254 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.61:35389
I20260929 11:29:59.746441 5320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.61:35389 every 8 connection(s)
I20260929 11:29:59.747889 5254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-1/data/info.pb
I20260929 11:29:59.752022 5321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:29:59.753016 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 5254
I20260929 11:29:59.753221 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-1/wal/instance
I20260929 11:29:59.753751 24164 external_mini_cluster.cc:1428] Setting key b3e6e7cd455314003799d6f6a892fa1f
I20260929 11:29:59.762385 5321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.153.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.60@KRBTEST.COM" created.
Sep 29 11:29:59 dist-test-slave-qk05 krb5kdc[5167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681399, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.61@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:29:59.789645 5324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.765418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.23.153.62:34561)
0929 11:29:59.765974 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:59.765980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:59.767131 (+ 1151us) server_negotiation.cc:407] Connection header received
0929 11:29:59.770105 (+ 2974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:59.770110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:59.770198 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:59.770381 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:59.772970 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:59.773795 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:59.776635 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:59.777121 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:59.777252 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:59.782117 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:59.782125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:59.782129 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:59.782194 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:59.784968 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:59.786155 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:59.786164 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:59.786167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:59.786537 (+ 370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:59.787258 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:59.787262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:59.787264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:59.787714 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:59.787866 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:59.788944 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0929 11:29:59.789207 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":270,"threads_started":1}
I20260929 11:29:59.790524 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.766006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:34561 (local address 127.0.0.1:55634)
0929 11:29:59.766891 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.766958 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 11:29:59.769627 (+ 2669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:29:59.770638 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:29:59.770653 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:29:59.771586 (+ 933us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:29:59.772682 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:59.772715 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:59.774045 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:59.774052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:59.776361 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:59.776395 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:59.777517 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:59.777530 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:59.777929 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:59.778813 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:29:59.778841 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:29:59.781735 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:29:59.785197 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:59.785209 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:59.785230 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:59.785963 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:59.786731 (+ 768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:59.786738 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:59.786742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:59.787112 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:59.787885 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:29:59.787896 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:29:59.788470 (+ 574us) client_negotiation.cc:769] Sending connection context
0929 11:29:59.788775 (+ 305us) client_negotiation.cc:240] Negotiation successful
0929 11:29:59.789225 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":330,"threads_started":1}
I20260929 11:29:59.792471 5321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:29:59 dist-test-slave-qk05 krb5kdc[5167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681399, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.61@KRBTEST.COM for kudu/127.23.153.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.60.keytab.
Entry for principal kudu/127.23.153.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.60.keytab.
I20260929 11:29:59.811065 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.60:37725
--webserver_interface=127.23.153.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.60.keytab
--principal=kudu/127.23.153.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:34561,127.23.153.61:35389,127.23.153.60:37725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:29:59.811558 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.793365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35389 (local address 127.0.0.1:54786)
0929 11:29:59.793586 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.793604 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:29:59.793713 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:29:59.794860 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:29:59.794865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:29:59.795354 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:29:59.795496 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:59.795506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:59.796902 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:59.796906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:59.798087 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:59.798103 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:59.799644 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:59.799651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:59.799906 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:59.800622 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:29:59.800645 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:29:59.804249 (+ 3604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:29:59.808531 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:59.808536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:59.808539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:59.809016 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:59.809501 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:59.809507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:59.809510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:59.809709 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:59.810502 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:29:59.810507 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:29:59.810667 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 11:29:59.810844 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 11:29:59.811215 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 11:29:59.811940 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.793680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54786 (local address 127.23.153.61:35389)
0929 11:29:59.794382 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:59.794389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:59.794414 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:29:59.794463 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:59.794469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:59.794674 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:59.794844 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:59.795665 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:59.796750 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:59.798273 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:59.799378 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:59.799591 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:59.804742 (+ 5151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:59.804762 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:59.804779 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:59.804821 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:59.808301 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:59.809189 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:59.809197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:59.809199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:59.809358 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:59.809844 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:59.809849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:59.809852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:59.810339 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:59.810528 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:59.811047 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 11:29:59.811474 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":525,"thread_start_us":392,"threads_started":1}
I20260929 11:29:59.814435 5321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:29:59.815708 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.815152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:54844)
0929 11:29:59.815344 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.815512 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 11:29:59.816174 5273 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
I20260929 11:29:59.818531 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.818016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:54856)
0929 11:29:59.818240 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.818358 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260929 11:29:59.818842 5321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:37725: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
I20260929 11:29:59.868253 5321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } attempt: 1
I20260929 11:29:59.869580 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.869017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:54872)
0929 11:29:59.869296 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.869421 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260929 11:29:59.870723 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.870349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:54882)
0929 11:29:59.870512 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.870601 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 11:29:59.870942 5321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:37725: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
I20260929 11:29:59.882786 5247 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } attempt: 5
Sep 29 11:29:59 dist-test-slave-qk05 krb5kdc[5167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681399, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for kudu/127.23.153.61@KRBTEST.COM
I20260929 11:29:59.898270 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.883808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54802 (local address 127.23.153.61:35389)
0929 11:29:59.884006 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 11:29:59.884012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:29:59.884037 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:29:59.884106 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:29:59.884111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:29:59.884208 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:29:59.884337 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:29:59.885154 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:59.886233 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:59.887940 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:29:59.888424 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:29:59.888557 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:59.892071 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:29:59.892077 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:29:59.892080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:29:59.892120 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:29:59.895377 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:59.896204 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:59.896210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:59.896212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:59.896360 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:29:59.896755 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:29:59.896768 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:29:59.896770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:29:59.897092 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:29:59.897240 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 11:29:59.897673 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 11:29:59.897927 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 11:29:59.898424 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.883666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35389 (local address 127.0.0.1:54802)
0929 11:29:59.883873 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.883891 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:29:59.884004 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:29:59.884359 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:29:59.884362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:29:59.884755 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:29:59.884964 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:59.884973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:59.886566 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:59.886572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:59.887726 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:29:59.887747 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:29:59.888601 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:29:59.888607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:29:59.888828 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:29:59.889542 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:29:59.889562 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:29:59.891819 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:29:59.895622 (+ 3803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:59.895627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:59.895630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:59.896067 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:59.896485 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:29:59.896489 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:29:59.896491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:29:59.896645 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:29:59.897222 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:29:59.897227 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:29:59.897398 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 11:29:59.897536 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 11:29:59.897923 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 11:29:59.900241 5247 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:29:59.901763 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.901072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:54892)
0929 11:29:59.901265 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.901513 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 11:29:59.903200 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.902721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:54896)
0929 11:29:59.902874 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.902990 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 11:29:59.903496 5247 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:37725: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
I20260929 11:29:59.939206 5321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } attempt: 2
I20260929 11:29:59.940477 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.939985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:54906)
0929 11:29:59.940221 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.940312 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 11:29:59.941916 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.941303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:54918)
0929 11:29:59.941692 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.941777 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267}
W20260929 11:29:59.942174 5321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:37725: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
I20260929 11:29:59.954762 5247 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } attempt: 1
I20260929 11:29:59.956061 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.955570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:54934)
0929 11:29:59.955775 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.955877 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 11:29:59.957036 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:29:59.956686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:54936)
0929 11:29:59.956833 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 11:29:59.956911 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 11:29:59.957257 5247 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:37725: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
I20260929 11:30:00.025528 5247 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } attempt: 2
I20260929 11:30:00.027081 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.026290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:54940)
0929 11:30:00.026733 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:00.026837 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260929 11:30:00.028213 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.027833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:54948)
0929 11:30:00.027998 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:00.028082 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 11:30:00.028461 5247 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:37725: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
I20260929 11:30:00.033524 5321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } attempt: 3
I20260929 11:30:00.035148 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.034533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:54952)
0929 11:30:00.034805 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:00.034903 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 11:30:00.036351 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.035867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:54968)
0929 11:30:00.036106 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:00.036196 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260929 11:30:00.036586 5321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:37725: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
I20260929 11:30:00.134701 5247 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } attempt: 3
I20260929 11:30:00.135864 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.135400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:54970)
0929 11:30:00.135590 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:00.135690 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 11:30:00.136873 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.136495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:54980)
0929 11:30:00.136670 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:00.136752 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 11:30:00.137082 5247 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:37725: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
I20260929 11:30:00.179889 5321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } attempt: 4
I20260929 11:30:00.181499 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.180952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:54996)
0929 11:30:00.181197 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:00.181301 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 11:30:00.182569 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.182210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:55006)
0929 11:30:00.182374 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:00.182450 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 11:30:00.182785 5321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:37725: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
W20260929 11:30:00.197991 5329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:00.198508 5329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:00.198670 5329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:00.206555 5329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:30:00.206712 5329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:00.206781 5329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:00.206856 5329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:30:00.206916 5329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:30:00.218439 5329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:34561,127.23.153.61:35389,127.23.153.60:37725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.60.keytab
--principal=kudu/127.23.153.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.60:37725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.153.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:00.220664 5329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:00.222214 5329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:00.235285 5335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:00.235355 5334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:00.235321 5337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:00.236115 5329 server_base.cc:1034] running on GCE node
I20260929 11:30:00.236977 5329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:00.239269 5329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:00.240790 5329 hybrid_clock.cc:648] HybridClock initialized: now 1790681400240702 us; error 56 us; skew 500 ppm
I20260929 11:30:00.241478 5329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:00 dist-test-slave-qk05 krb5kdc[5167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681400, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:00.250672 5329 init.cc:377] Logged in from keytab as kudu/127.23.153.60@KRBTEST.COM (short username kudu)
I20260929 11:30:00.252789 5329 webserver.cc:460] Webserver started at http://127.23.153.60:46679/ using document root <none> and password file <none>
I20260929 11:30:00.253666 5329 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:00.253813 5329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:00.254150 5329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:30:00.257762 5329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-2/data/instance:
uuid: "aba5c4aff873496a9ee89d6d18616f97"
format_stamp: "Formatted at 2026-09-29 11:30:00 on dist-test-slave-qk05"
server_key: "0ea3cd9141799f2a2c45563375e36d11"
server_key_iv: "bd5bcedbecad507938d5badeecc28470"
server_key_version: "encryptionkey@0"
I20260929 11:30:00.258720 5329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-2/wal/instance:
uuid: "aba5c4aff873496a9ee89d6d18616f97"
format_stamp: "Formatted at 2026-09-29 11:30:00 on dist-test-slave-qk05"
server_key: "0ea3cd9141799f2a2c45563375e36d11"
server_key_iv: "bd5bcedbecad507938d5badeecc28470"
server_key_version: "encryptionkey@0"
I20260929 11:30:00.265336 5329 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260929 11:30:00.270416 5344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:00.272246 5329 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 11:30:00.272476 5329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-2/wal
uuid: "aba5c4aff873496a9ee89d6d18616f97"
format_stamp: "Formatted at 2026-09-29 11:30:00 on dist-test-slave-qk05"
server_key: "0ea3cd9141799f2a2c45563375e36d11"
server_key_iv: "bd5bcedbecad507938d5badeecc28470"
server_key_version: "encryptionkey@0"
I20260929 11:30:00.272707 5329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:00.298462 5329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:00.303686 5329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:00.303993 5329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:00.314301 5247 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } attempt: 4
I20260929 11:30:00.315721 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.315128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:55016)
0929 11:30:00.315398 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:00.315524 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260929 11:30:00.317001 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.316528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:55032)
0929 11:30:00.316720 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:00.316815 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 11:30:00.317260 5247 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:37725: Network error: Client connection negotiation failed: client connection to 127.23.153.60:37725: connect: Connection refused (error 111)
I20260929 11:30:00.325757 5329 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.60:37725
I20260929 11:30:00.325795 5395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.60:37725 every 8 connection(s)
I20260929 11:30:00.327322 5329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-2/data/info.pb
I20260929 11:30:00.327826 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 5329
I20260929 11:30:00.327988 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790681181154781-24164-0/minicluster-data/master-2/wal/instance
I20260929 11:30:00.328596 24164 external_mini_cluster.cc:1428] Setting key 2489e7bb6b53b500066f7c195fc9473b
I20260929 11:30:00.332515 5396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:30:00.341629 5396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:30:00 dist-test-slave-qk05 krb5kdc[5167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.60@KRBTEST.COM
Sep 29 11:30:00 dist-test-slave-qk05 krb5kdc[5167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681400, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.60@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:30:00.355786 5399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.331421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.23.153.60:37725)
0929 11:30:00.332290 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:00.332307 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:00.332452 (+ 145us) server_negotiation.cc:407] Connection header received
0929 11:30:00.334211 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:00.334235 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:00.334676 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:00.335143 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:00.336719 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.338771 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.340481 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.341257 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.341503 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:00.347081 (+ 5578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:00.347096 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:00.347119 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:00.347172 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:00.350764 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.351875 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.351886 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.351896 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.352161 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.352618 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.352622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.352624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.353250 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:00.353505 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:00.354006 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 11:30:00.354413 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":406,"threads_started":1}
I20260929 11:30:00.362535 5400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.344814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:34561 (local address 127.0.0.1:55650)
0929 11:30:00.345934 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:00.345991 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:00.346269 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:00.347175 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:00.347181 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:00.347925 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:30:00.348283 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:00.348294 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.349874 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.349879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:00.351251 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:00.351268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.352142 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.352147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:00.352309 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:00.352990 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:30:00.353013 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:30:00.355684 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:30:00.359627 (+ 3943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:00.359641 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:00.359648 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:00.360235 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:00.360648 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:00.360654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:00.360657 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:00.360794 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:00.361349 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:30:00.361358 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:30:00.361559 (+ 201us) client_negotiation.cc:769] Sending connection context
0929 11:30:00.361728 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 11:30:00.362105 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":631,"threads_started":1}
I20260929 11:30:00.362577 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.344930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.23.153.62:34561)
0929 11:30:00.346743 (+ 1813us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:00.346747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:00.346764 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:30:00.346838 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:00.346842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:00.346942 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:00.347113 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:00.348528 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.349657 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.351460 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.351941 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.352077 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:00.355959 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:00.355967 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:00.355972 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:00.356031 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:00.359462 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.360369 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.360375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.360378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.360528 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.360913 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.360918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.360920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.361213 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:00.361367 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:00.361889 (+ 522us) server_negotiation.cc:299] Negotiation successful
0929 11:30:00.362138 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1698,"thread_start_us":1591,"threads_started":1}
I20260929 11:30:00.364203 5396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:30:00 dist-test-slave-qk05 krb5kdc[5167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
Sep 29 11:30:00 dist-test-slave-qk05 krb5kdc[5167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681400, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.60@KRBTEST.COM for kudu/127.23.153.61@KRBTEST.COM
I20260929 11:30:00.376396 5402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.359603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.23.153.62:34561)
0929 11:30:00.360170 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:00.360175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:00.360196 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:30:00.360256 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:00.360260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:00.360320 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:00.360420 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:00.361691 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.362597 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.364116 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.364596 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.364717 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:00.370198 (+ 5481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:00.370204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:00.370208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:00.370250 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:00.373244 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.374213 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.374218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.374220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.374365 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.374793 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.374797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.374800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.375174 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:00.375444 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:00.375774 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 11:30:00.376035 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":266,"threads_started":1}
I20260929 11:30:00.380199 5400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.365463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35389 (local address 127.0.0.1:54808)
0929 11:30:00.365820 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:00.365842 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:00.366114 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:00.366561 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:00.366567 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:00.367122 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:30:00.367363 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:00.367376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.368357 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.368363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:00.369521 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:00.369540 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.370312 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.370318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:00.370527 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:00.371114 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:30:00.371139 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:30:00.373452 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:30:00.376765 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:00.376770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:00.376774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:00.377321 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:00.377947 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:00.377952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:00.377955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:00.378153 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:00.379408 (+ 1255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:30:00.379413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:30:00.379586 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 11:30:00.379709 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 11:30:00.379944 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220}
I20260929 11:30:00.380484 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.365429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54808 (local address 127.23.153.61:35389)
0929 11:30:00.365646 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:00.365650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:00.365996 (+ 346us) server_negotiation.cc:407] Connection header received
0929 11:30:00.366236 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:00.366241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:00.366305 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:00.366388 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:00.367519 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.368200 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.369684 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.370126 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.370253 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:00.373690 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:00.373697 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:00.373702 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:00.373758 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:00.376581 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.377527 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.377533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.377536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.377700 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.378861 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.378866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.378869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.379257 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:00.379453 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:00.379848 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 11:30:00.380111 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:30:00.381748 5396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:30:00 dist-test-slave-qk05 krb5kdc[5167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.61@KRBTEST.COM
Sep 29 11:30:00 dist-test-slave-qk05 krb5kdc[5167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681400, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.60@KRBTEST.COM for kudu/127.23.153.60@KRBTEST.COM
I20260929 11:30:00.395355 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.378368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.23.153.61:35389)
0929 11:30:00.378948 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:00.378952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:00.378969 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:30:00.379067 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:00.379073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:00.379141 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:00.379223 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:00.380885 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.381879 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.383532 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.384103 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.384238 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:00.388966 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:00.388973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:00.388979 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:00.389026 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:00.392222 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.393183 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.393190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.393193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.393364 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.393824 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.393830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.393834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.394132 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:00.394405 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:00.394668 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 11:30:00.394890 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":248,"threads_started":1}
I20260929 11:30:00.395344 5400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.382488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:55046)
0929 11:30:00.382764 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:00.382789 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:00.382876 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:00.383179 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:00.383182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:00.383564 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:30:00.383706 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:00.383715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.384851 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.384856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:00.385745 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:00.385758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.386523 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.386529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:00.386692 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:00.387264 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:30:00.387284 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:30:00.389784 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:30:00.392756 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:00.392761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:00.392763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:00.393162 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:00.393595 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:00.393598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:00.393600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:00.393725 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:00.394246 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:30:00.394251 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:30:00.394405 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 11:30:00.394530 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 11:30:00.394883 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260929 11:30:00.395586 5399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.382687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55046 (local address 127.23.153.60:37725)
0929 11:30:00.382865 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:00.382871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:00.382890 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:30:00.382964 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:00.382969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:00.383071 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:00.383164 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:00.383850 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.384713 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.385917 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.386348 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.386464 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:00.389963 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:00.389968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:00.389971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:00.390003 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:00.392595 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.393304 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.393308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.393310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.393468 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.393834 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.393838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.393840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.394124 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:00.394261 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:00.394643 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 11:30:00.395125 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":60}
I20260929 11:30:00.400556 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:30:00.401373 5396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97: Bootstrap starting.
I20260929 11:30:00.404990 5396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97: Neither blocks nor log segments found. Creating new log.
I20260929 11:30:00.406553 5396 log.cc:824] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:00.425750 5402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.405474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.23.153.62:34561)
0929 11:30:00.405657 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:00.405661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:00.406358 (+ 697us) server_negotiation.cc:407] Connection header received
0929 11:30:00.406674 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:00.406680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:00.406766 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:00.406858 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:00.409405 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.410542 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.412852 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.413486 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.413594 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:00.418488 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:00.418497 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:00.418503 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:00.418557 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:00.422470 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.423404 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.423410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.423414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.423576 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.424031 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.424036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.424040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.424413 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:00.424643 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:00.425139 (+ 496us) server_negotiation.cc:299] Negotiation successful
0929 11:30:00.425403 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:30:00.426103 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.405377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.23.153.61:35389)
0929 11:30:00.405593 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:00.405600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:00.406687 (+ 1087us) server_negotiation.cc:407] Connection header received
0929 11:30:00.406961 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:00.406966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:00.407083 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:00.407154 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:00.409612 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.410612 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.413015 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.413653 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.413744 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:00.418166 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:00.418173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:00.418178 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:00.418229 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:00.422018 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.423112 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.423119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.423123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.423316 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.423760 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.423765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.423768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.424143 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:00.424389 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:00.425516 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0929 11:30:00.425774 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 11:30:00.430327 5396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97: No bootstrap required, opened a new log
I20260929 11:30:00.434376 5399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.406951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.23.153.60:37725)
0929 11:30:00.407186 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:00.407193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:00.414313 (+ 7120us) server_negotiation.cc:407] Connection header received
0929 11:30:00.414837 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:00.414844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:00.414936 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:00.415080 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:00.417606 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.419771 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.422042 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.422896 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.423058 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:00.427442 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:00.427448 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:00.427451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:00.427491 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:00.431292 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.432166 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.432171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.432174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.432348 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.432768 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.432772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.432774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.433125 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:00.433398 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:00.433752 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 11:30:00.434009 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 11:30:00.439476 5396 raft_consensus.cc:348] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } }
I20260929 11:30:00.439865 5396 raft_consensus.cc:374] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:30:00.439990 5396 raft_consensus.cc:729] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba5c4aff873496a9ee89d6d18616f97, State: Initialized, Role: FOLLOWER
I20260929 11:30:00.440701 5396 consensus_queue.cc:260] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } }
I20260929 11:30:00.441748 5412 sys_catalog.cc:455] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } } }
I20260929 11:30:00.442196 5412 sys_catalog.cc:458] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:30:00.443411 5396 sys_catalog.cc:564] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 11:30:00.453873 5423 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:30:00.454027 5423 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:30:00.450862 5329 master_runner.cc:186] Error getting master registration for 127.23.153.62:34561: OK, instance_id { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" instance_seqno: 1790681399160222 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 11:30:00.462797 5329 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 11:30:00.464212 5412 raft_consensus.cc:482] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:30:00.464531 5412 raft_consensus.cc:504] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } }
I20260929 11:30:00.466409 5412 leader_election.cc:288] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 141b13b505fe40c497a3d133d0067aae (127.23.153.62:34561), 9552f26c9c9a40deadcbd25184ee4e03 (127.23.153.61:35389)
I20260929 11:30:00.481949 5321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } attempt: 5
Sep 29 11:30:00 dist-test-slave-qk05 krb5kdc[5167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681399, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.61@KRBTEST.COM for kudu/127.23.153.60@KRBTEST.COM
I20260929 11:30:00.502060 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.483088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:55058)
0929 11:30:00.483899 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:00.483920 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:00.484046 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:00.484637 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:00.484650 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:00.485201 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:30:00.485455 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:00.485469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.486925 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.486944 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:00.488464 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:00.488488 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.489343 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.489349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:00.489587 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:00.490277 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:30:00.490304 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:30:00.493516 (+ 3212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:30:00.498798 (+ 5282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:00.498804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:00.498809 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:00.499442 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:00.499979 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:00.499985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:00.499988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:00.500167 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:00.500867 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:30:00.500873 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:30:00.501061 (+ 188us) client_negotiation.cc:769] Sending connection context
0929 11:30:00.501212 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 11:30:00.501592 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676}
I20260929 11:30:00.503320 5399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.483143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.23.153.60:37725)
0929 11:30:00.484290 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:00.484296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:00.484314 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:30:00.484368 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:00.484377 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:00.484499 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:00.484595 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:00.485619 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.486638 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.488685 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.489156 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.489295 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:00.493787 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:00.493795 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:00.493800 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:00.493857 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:00.498636 (+ 4779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.499615 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.499625 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.499628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.499821 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.500295 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.500307 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.500311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.500724 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:00.500882 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:00.502475 (+ 1593us) server_negotiation.cc:299] Negotiation successful
0929 11:30:00.502867 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1013}
I20260929 11:30:00.510856 5321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03: Bootstrap starting.
I20260929 11:30:00.515416 5321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03: Neither blocks nor log segments found. Creating new log.
I20260929 11:30:00.516594 5321 log.cc:824] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:00.520170 5321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03: No bootstrap required, opened a new log
I20260929 11:30:00.529696 5321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } }
I20260929 11:30:00.530112 5321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:30:00.530226 5321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9552f26c9c9a40deadcbd25184ee4e03, State: Initialized, Role: FOLLOWER
I20260929 11:30:00.530865 5321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } }
I20260929 11:30:00.531895 5426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } } }
I20260929 11:30:00.532334 5426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:30:00.533159 5321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:30:00.533263 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aba5c4aff873496a9ee89d6d18616f97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9552f26c9c9a40deadcbd25184ee4e03" is_pre_election: true
I20260929 11:30:00.533958 5296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aba5c4aff873496a9ee89d6d18616f97 in term 0.
I20260929 11:30:00.534871 5348 leader_election.cc:302] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9552f26c9c9a40deadcbd25184ee4e03, aba5c4aff873496a9ee89d6d18616f97; no voters:
I20260929 11:30:00.535429 5412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:30:00.535593 5412 raft_consensus.cc:482] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:30:00.535694 5412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:30:00.539222 5412 raft_consensus.cc:504] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } }
I20260929 11:30:00.542722 5412 leader_election.cc:288] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [CANDIDATE]: Term 1 election: Requested vote from peers 141b13b505fe40c497a3d133d0067aae (127.23.153.62:34561), 9552f26c9c9a40deadcbd25184ee4e03 (127.23.153.61:35389)
I20260929 11:30:00.543781 5296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aba5c4aff873496a9ee89d6d18616f97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9552f26c9c9a40deadcbd25184ee4e03"
I20260929 11:30:00.544049 5296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:30:00.548592 5296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aba5c4aff873496a9ee89d6d18616f97 in term 1.
I20260929 11:30:00.549226 5348 leader_election.cc:302] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9552f26c9c9a40deadcbd25184ee4e03, aba5c4aff873496a9ee89d6d18616f97; no voters:
I20260929 11:30:00.549633 5412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:30:00.550298 5412 raft_consensus.cc:686] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [term 1 LEADER]: Becoming Leader. State: Replica: aba5c4aff873496a9ee89d6d18616f97, State: Running, Role: LEADER
I20260929 11:30:00.550973 5412 consensus_queue.cc:237] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } }
W20260929 11:30:00.558625 5350 tablet.cc:2369] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:30:00.569538 5435 sys_catalog.cc:455] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aba5c4aff873496a9ee89d6d18616f97. Latest consensus state: current_term: 1 leader_uuid: "aba5c4aff873496a9ee89d6d18616f97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } } }
I20260929 11:30:00.570061 5435 sys_catalog.cc:458] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [sys.catalog]: This master's current role is: LEADER
W20260929 11:30:00.579939 5439 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:30:00.580209 5439 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:30:00.581473 5441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:30:00.567564 5254 master_runner.cc:186] Error getting master registration for 127.23.153.62:34561: OK, instance_id { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" instance_seqno: 1790681399160222 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 11:30:00.584554 5254 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 11:30:00.587085 5441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:30:00.594501 5247 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } attempt: 5
Sep 29 11:30:00 dist-test-slave-qk05 krb5kdc[5167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681399, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for kudu/127.23.153.60@KRBTEST.COM
I20260929 11:30:00.612208 5296 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03 [term 1 FOLLOWER]: Refusing update from remote peer aba5c4aff873496a9ee89d6d18616f97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:30:00.613437 5435 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:30:00.634354 5399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.596170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55068 (local address 127.23.153.60:37725)
0929 11:30:00.596362 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:00.596368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:00.596386 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:30:00.596449 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:00.596455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:00.596534 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:00.596638 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:00.597923 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.599532 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.601923 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.602743 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.602914 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:00.613054 (+ 10140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:00.613062 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:00.613067 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:00.613121 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:00.619614 (+ 6493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.622085 (+ 2471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.622094 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.622098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.622745 (+ 647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:00.624212 (+ 1467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:00.624224 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:00.624228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:00.625493 (+ 1265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:00.626789 (+ 1296us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:00.627897 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0929 11:30:00.632284 (+ 4387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 11:30:00.634981 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:00.595317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:37725 (local address 127.0.0.1:55068)
0929 11:30:00.595757 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:00.595778 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:00.595928 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:00.596843 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:00.596849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:00.597443 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:30:00.597711 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:00.597727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.599817 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.599825 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:00.601597 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:00.601627 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:00.603334 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:00.603343 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:00.603574 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:00.604584 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:30:00.604650 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:30:00.612663 (+ 8013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:30:00.620045 (+ 7382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:00.620052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:00.620057 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:00.621699 (+ 1642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:00.623127 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:00.623134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:00.623138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:00.623587 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:00.626383 (+ 2796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:30:00.626394 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:30:00.627064 (+ 670us) client_negotiation.cc:769] Sending connection context
0929 11:30:00.631325 (+ 4261us) client_negotiation.cc:240] Negotiation successful
0929 11:30:00.633013 (+ 1688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302}
I20260929 11:30:00.641503 5426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aba5c4aff873496a9ee89d6d18616f97. Latest consensus state: current_term: 1 leader_uuid: "aba5c4aff873496a9ee89d6d18616f97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } } }
I20260929 11:30:00.641875 5426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:30:00.644661 5247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae: Bootstrap starting.
I20260929 11:30:00.651345 5412 sys_catalog.cc:455] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aba5c4aff873496a9ee89d6d18616f97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } } }
I20260929 11:30:00.651810 5412 sys_catalog.cc:458] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [sys.catalog]: This master's current role is: LEADER
I20260929 11:30:00.657275 5247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae: Neither blocks nor log segments found. Creating new log.
I20260929 11:30:00.662842 5441 catalog_manager.cc:1347] Generated new cluster ID: a151866e128d4b1f9d58548198d10d24
I20260929 11:30:00.662979 5441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:30:00.663576 5426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aba5c4aff873496a9ee89d6d18616f97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } } }
I20260929 11:30:00.663882 5426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:30:00.664872 5247 log.cc:824] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:00.681169 5247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae: No bootstrap required, opened a new log
I20260929 11:30:00.694336 5441 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:30:00.699498 5441 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:30:00.710989 5247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } }
I20260929 11:30:00.711946 5247 raft_consensus.cc:374] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:30:00.712208 5247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 141b13b505fe40c497a3d133d0067aae, State: Initialized, Role: FOLLOWER
I20260929 11:30:00.713965 5247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } }
I20260929 11:30:00.722179 5441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97: Generated new TSK 0
I20260929 11:30:00.724300 5441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:30:00.727892 5222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aba5c4aff873496a9ee89d6d18616f97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "141b13b505fe40c497a3d133d0067aae" is_pre_election: true
I20260929 11:30:00.728478 5220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:30:00.728952 5221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aba5c4aff873496a9ee89d6d18616f97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "141b13b505fe40c497a3d133d0067aae"
I20260929 11:30:00.731215 5247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:30:00.758641 5220 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae [term 1 FOLLOWER]: Refusing update from remote peer aba5c4aff873496a9ee89d6d18616f97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:30:00.768378 5435 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:30:00.768437 5448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } } }
I20260929 11:30:00.768901 5448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:30:00.812114 5448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae [sys.catalog]: SysCatalogTable state changed. Reason: New leader aba5c4aff873496a9ee89d6d18616f97. Latest consensus state: current_term: 1 leader_uuid: "aba5c4aff873496a9ee89d6d18616f97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } } }
I20260929 11:30:00.812474 5448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:30:00.813202 5456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aba5c4aff873496a9ee89d6d18616f97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } } }
I20260929 11:30:00.813817 5456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:30:00.834409 5469 catalog_manager.cc:1259] Loaded cluster ID: a151866e128d4b1f9d58548198d10d24
I20260929 11:30:00.834604 5469 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae: loading cluster ID for follower catalog manager: success
I20260929 11:30:00.838572 5469 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae: acquiring CA information for follower catalog manager: success
I20260929 11:30:00.841414 5469 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 141b13b505fe40c497a3d133d0067aae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 11:30:01.196842 5435 sys_catalog.cc:455] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aba5c4aff873496a9ee89d6d18616f97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "141b13b505fe40c497a3d133d0067aae" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34561 } } peers { permanent_uuid: "9552f26c9c9a40deadcbd25184ee4e03" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35389 } } peers { permanent_uuid: "aba5c4aff873496a9ee89d6d18616f97" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 37725 } } }
I20260929 11:30:01.197170 5435 sys_catalog.cc:458] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 [sys.catalog]: This master's current role is: LEADER
I20260929 11:30:01.582332 5439 catalog_manager.cc:1259] Loaded cluster ID: a151866e128d4b1f9d58548198d10d24
I20260929 11:30:01.582463 5439 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03: loading cluster ID for follower catalog manager: success
I20260929 11:30:01.585093 5439 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03: acquiring CA information for follower catalog manager: success
I20260929 11:30:01.587234 5439 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9552f26c9c9a40deadcbd25184ee4e03: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 11:30:01.889513 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 5180
I20260929 11:30:01.890760 5242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:02.008296 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 5180
I20260929 11:30:02.029028 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 5254
I20260929 11:30:02.030479 5316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:02.159338 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 5254
I20260929 11:30:02.165552 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:02.164755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:34561 (local address 127.0.0.1:55680)
0929 11:30:02.165237 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:02.165362 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":269,"threads_started":1}
W20260929 11:30:02.165908 5347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.153.62:34561: connect: Connection refused (error 111)
I20260929 11:30:02.168272 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:02.167686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:34561 (local address 127.0.0.1:55682)
0929 11:30:02.167925 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:02.168078 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260929 11:30:02.168706 5347 consensus_peers.cc:487] T 00000000000000000000000000000000 P aba5c4aff873496a9ee89d6d18616f97 -> Peer 141b13b505fe40c497a3d133d0067aae (127.23.153.62:34561): Couldn't send request to peer 141b13b505fe40c497a3d133d0067aae. Status: Network error: Client connection negotiation failed: client connection to 127.23.153.62:34561: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 11:30:02.182269 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 5329
I20260929 11:30:02.184064 5391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:02.306247 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 5329
2026-09-29T11:30:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4891 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:30:02 dist-test-slave-qk05 krb5kdc[5479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:30:02 dist-test-slave-qk05 krb5kdc[5479](info): set up 2 sockets
Sep 29 11:30:02 dist-test-slave-qk05 krb5kdc[5479](info): commencing operation
krb5kdc: starting...
W20260929 11:30:03.375710 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:30:03 dist-test-slave-qk05 krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:30:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:30:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:30:03.639892 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:43945
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:39849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:43945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:30:04.019380 5492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:04.019721 5492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:04.019874 5492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:04.029032 5492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:30:04.029191 5492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:04.029246 5492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:04.029332 5492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:30:04.029389 5492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:30:04.041348 5492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:39849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:43945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:43945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:04.043546 5492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:04.045239 5492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:04.059970 5497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:04.060025 5498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:04.059939 5500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:04.061110 5492 server_base.cc:1034] running on GCE node
I20260929 11:30:04.062029 5492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:04.063800 5492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:04.065076 5492 hybrid_clock.cc:648] HybridClock initialized: now 1790681404065018 us; error 43 us; skew 500 ppm
I20260929 11:30:04.065523 5492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:04 dist-test-slave-qk05 krb5kdc[5479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681404, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:04.069767 5492 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:30:04.072305 5492 webserver.cc:460] Webserver started at http://127.23.153.62:43883/ using document root <none> and password file <none>
I20260929 11:30:04.073383 5492 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:04.073770 5492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:04.074177 5492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:30:04.077859 5492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "9317854b7cf1463781a0d210540f4114"
format_stamp: "Formatted at 2026-09-29 11:30:04 on dist-test-slave-qk05"
server_key: "4a089a89b8b5a427da6d7976bc4c4bc3"
server_key_iv: "fc78748e8e49d7a71b3285ba441029c8"
server_key_version: "encryptionkey@0"
I20260929 11:30:04.078785 5492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "9317854b7cf1463781a0d210540f4114"
format_stamp: "Formatted at 2026-09-29 11:30:04 on dist-test-slave-qk05"
server_key: "4a089a89b8b5a427da6d7976bc4c4bc3"
server_key_iv: "fc78748e8e49d7a71b3285ba441029c8"
server_key_version: "encryptionkey@0"
I20260929 11:30:04.085295 5492 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:30:04.089804 5507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:04.091688 5492 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 11:30:04.091934 5492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "9317854b7cf1463781a0d210540f4114"
format_stamp: "Formatted at 2026-09-29 11:30:04 on dist-test-slave-qk05"
server_key: "4a089a89b8b5a427da6d7976bc4c4bc3"
server_key_iv: "fc78748e8e49d7a71b3285ba441029c8"
server_key_version: "encryptionkey@0"
I20260929 11:30:04.092155 5492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:04.124255 5492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:04.130554 5492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:04.130913 5492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:04.151338 5492 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:43945
I20260929 11:30:04.151346 5558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:43945 every 8 connection(s)
I20260929 11:30:04.152842 5492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:30:04.157270 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:30:04.159189 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 5492
I20260929 11:30:04.159404 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:30:04.160377 24164 external_mini_cluster.cc:1428] Setting key 6022b0a3929f8e0df047535c966661e9
I20260929 11:30:04.171168 5559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114: Bootstrap starting.
I20260929 11:30:04.174733 5559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114: Neither blocks nor log segments found. Creating new log.
I20260929 11:30:04.175976 5559 log.cc:824] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114: Log is configured to *not* fsync() on all Append() calls
Sep 29 11:30:04 dist-test-slave-qk05 krb5kdc[5479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:30:04.180536 5559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114: No bootstrap required, opened a new log
I20260929 11:30:04.187943 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:04.162631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46908 (local address 127.23.153.62:43945)
0929 11:30:04.164089 (+ 1458us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:04.164108 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:04.164160 (+ 52us) server_negotiation.cc:407] Connection header received
0929 11:30:04.166039 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:04.166066 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:04.166565 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:04.167053 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:04.169179 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:04.170648 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:04.172643 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:04.173704 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:04.173963 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:04.179717 (+ 5754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:04.179726 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:04.179744 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:04.179796 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:04.182977 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:04.183876 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:04.183884 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:04.183892 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:04.184096 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:04.184519 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:04.184523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:04.184525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:04.185178 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:04.185534 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:04.186041 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 11:30:04.186547 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1170,"thread_start_us":644,"threads_started":1}
I20260929 11:30:04.189522 5559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9317854b7cf1463781a0d210540f4114" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 43945 } }
I20260929 11:30:04.190065 5559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:30:04.190189 5559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9317854b7cf1463781a0d210540f4114, State: Initialized, Role: FOLLOWER
I20260929 11:30:04.190861 5559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9317854b7cf1463781a0d210540f4114" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 43945 } }
I20260929 11:30:04.191223 5559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:30:04.191373 5559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:30:04.191555 5559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:30:04.192838 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:30:04.194824 5559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9317854b7cf1463781a0d210540f4114" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 43945 } }
I20260929 11:30:04.195574 5559 leader_election.cc:302] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9317854b7cf1463781a0d210540f4114; no voters:
I20260929 11:30:04.196369 5559 leader_election.cc:288] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:30:04.196533 5564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:30:04.197795 5564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114 [term 1 LEADER]: Becoming Leader. State: Replica: 9317854b7cf1463781a0d210540f4114, State: Running, Role: LEADER
I20260929 11:30:04.198377 5564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9317854b7cf1463781a0d210540f4114" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 43945 } }
I20260929 11:30:04.198714 5559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:30:04.201292 5566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9317854b7cf1463781a0d210540f4114. Latest consensus state: current_term: 1 leader_uuid: "9317854b7cf1463781a0d210540f4114" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9317854b7cf1463781a0d210540f4114" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 43945 } } }
I20260929 11:30:04.201722 5566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114 [sys.catalog]: This master's current role is: LEADER
I20260929 11:30:04.202250 5565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9317854b7cf1463781a0d210540f4114" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9317854b7cf1463781a0d210540f4114" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 43945 } } }
I20260929 11:30:04.202593 5565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114 [sys.catalog]: This master's current role is: LEADER
I20260929 11:30:04.204514 5572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:30:04.210283 5572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:30:04.222100 5572 catalog_manager.cc:1347] Generated new cluster ID: 6ce6a5c4519845bf9dfdfc1f2b61a5b4
I20260929 11:30:04.222239 5572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:30:04.242365 5572 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:30:04.244217 5572 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:30:04.260526 5572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9317854b7cf1463781a0d210540f4114: Generated new TSK 0
I20260929 11:30:04.261511 5572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:30:04.307471 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 5492
I20260929 11:30:04.309031 5554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:04.414974 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 5492
2026-09-29T11:30:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2105 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:30:04 dist-test-slave-qk05 krb5kdc[5587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:30:04 dist-test-slave-qk05 krb5kdc[5587](info): set up 2 sockets
Sep 29 11:30:04 dist-test-slave-qk05 krb5kdc[5587](info): commencing operation
krb5kdc: starting...
W20260929 11:30:05.463483 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:30:05 dist-test-slave-qk05 krb5kdc[5587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:30:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:30:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:30:05.706797 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:44055
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:44055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:30:06.074898 5600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:06.075300 5600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:06.075445 5600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:06.083889 5600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:30:06.084051 5600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:06.084133 5600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:06.084210 5600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:30:06.084273 5600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:30:06.095696 5600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:44055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:44055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:06.097150 5600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:06.098567 5600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:06.110883 5605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:06.110829 5606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:06.110839 5608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:06.111994 5600 server_base.cc:1034] running on GCE node
I20260929 11:30:06.112905 5600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:06.114528 5600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:06.115798 5600 hybrid_clock.cc:648] HybridClock initialized: now 1790681406115744 us; error 50 us; skew 500 ppm
I20260929 11:30:06.116220 5600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:06 dist-test-slave-qk05 krb5kdc[5587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681406, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:06.123476 5600 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:30:06.125344 5600 webserver.cc:460] Webserver started at http://127.23.153.62:39609/ using document root <none> and password file <none>
I20260929 11:30:06.126158 5600 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:06.126307 5600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:06.126694 5600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:30:06.130270 5600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "076d6020653b415c9439019df315f402"
format_stamp: "Formatted at 2026-09-29 11:30:06 on dist-test-slave-qk05"
server_key: "03ac5bb2b8d499ca591bec6422281109"
server_key_iv: "b76f1e61989ea99354821039a1af583e"
server_key_version: "encryptionkey@0"
I20260929 11:30:06.131286 5600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "076d6020653b415c9439019df315f402"
format_stamp: "Formatted at 2026-09-29 11:30:06 on dist-test-slave-qk05"
server_key: "03ac5bb2b8d499ca591bec6422281109"
server_key_iv: "b76f1e61989ea99354821039a1af583e"
server_key_version: "encryptionkey@0"
I20260929 11:30:06.137491 5600 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:30:06.141681 5615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:06.143391 5600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:30:06.143625 5600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "076d6020653b415c9439019df315f402"
format_stamp: "Formatted at 2026-09-29 11:30:06 on dist-test-slave-qk05"
server_key: "03ac5bb2b8d499ca591bec6422281109"
server_key_iv: "b76f1e61989ea99354821039a1af583e"
server_key_version: "encryptionkey@0"
I20260929 11:30:06.143847 5600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:06.177762 5600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:06.182749 5600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:06.183115 5600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:06.202333 5600 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:44055
I20260929 11:30:06.202364 5666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:44055 every 8 connection(s)
I20260929 11:30:06.203840 5600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:30:06.208002 5667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:30:06.213260 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 5600
I20260929 11:30:06.213629 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:30:06.214310 24164 external_mini_cluster.cc:1428] Setting key 2986719892feb3e07331c64e08023b23
I20260929 11:30:06.221993 5667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402: Bootstrap starting.
I20260929 11:30:06.225564 5667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402: Neither blocks nor log segments found. Creating new log.
I20260929 11:30:06.227087 5667 log.cc:824] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402: Log is configured to *not* fsync() on all Append() calls
Sep 29 11:30:06 dist-test-slave-qk05 krb5kdc[5587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:30:06.231853 5667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402: No bootstrap required, opened a new log
I20260929 11:30:06.242750 5667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076d6020653b415c9439019df315f402" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44055 } }
I20260929 11:30:06.243295 5667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:30:06.243434 5667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 076d6020653b415c9439019df315f402, State: Initialized, Role: FOLLOWER
I20260929 11:30:06.243597 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:06.216500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54770 (local address 127.23.153.62:44055)
0929 11:30:06.217398 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:06.217416 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:06.217483 (+ 67us) server_negotiation.cc:407] Connection header received
0929 11:30:06.220093 (+ 2610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:06.220134 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:06.220748 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:06.221268 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:06.222773 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:06.224282 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:06.225796 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:06.227054 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:06.227556 (+ 502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:06.232732 (+ 5176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:06.232746 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:06.232770 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:06.232855 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:06.237331 (+ 4476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:06.238410 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:06.238421 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:06.238432 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:06.238727 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:06.239285 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:06.239291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:06.239298 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:06.240197 (+ 899us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:06.240555 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:06.241469 (+ 914us) server_negotiation.cc:299] Negotiation successful
0929 11:30:06.242044 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":379,"threads_started":1}
I20260929 11:30:06.244150 5667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076d6020653b415c9439019df315f402" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44055 } }
I20260929 11:30:06.244426 5667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:30:06.244560 5667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:30:06.244735 5667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:30:06.247408 5667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076d6020653b415c9439019df315f402" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44055 } }
I20260929 11:30:06.248059 5667 leader_election.cc:302] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 076d6020653b415c9439019df315f402; no voters:
I20260929 11:30:06.248788 5667 leader_election.cc:288] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:30:06.248951 5672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:30:06.251402 5672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 1 LEADER]: Becoming Leader. State: Replica: 076d6020653b415c9439019df315f402, State: Running, Role: LEADER
I20260929 11:30:06.252193 5672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076d6020653b415c9439019df315f402" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44055 } }
I20260929 11:30:06.252436 5667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:30:06.255642 5673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "076d6020653b415c9439019df315f402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076d6020653b415c9439019df315f402" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44055 } } }
I20260929 11:30:06.256098 5673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [sys.catalog]: This master's current role is: LEADER
I20260929 11:30:06.256747 5674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 076d6020653b415c9439019df315f402. Latest consensus state: current_term: 1 leader_uuid: "076d6020653b415c9439019df315f402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076d6020653b415c9439019df315f402" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44055 } } }
I20260929 11:30:06.257130 5674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [sys.catalog]: This master's current role is: LEADER
I20260929 11:30:06.260255 5678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:30:06.266748 5678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:30:06.276420 5678 catalog_manager.cc:1347] Generated new cluster ID: 53863fcc9f3c46b2bb4eb3860d0f5679
I20260929 11:30:06.276566 5678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:30:06.307278 5678 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:30:06.309172 5678 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:30:06.326169 5678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402: Generated new TSK 0
I20260929 11:30:06.327410 5678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
I20260929 11:30:06.383462 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:44055
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:30:06.766312 5693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:06.766652 5693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:06.766753 5693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:06.774887 5693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:06.775086 5693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:06.775226 5693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:30:06.786654 5693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:44055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:06.788226 5693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:06.789747 5693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:06.805457 5699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:06.805970 5701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:06.805851 5698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:06.809171 5693 server_base.cc:1034] running on GCE node
I20260929 11:30:06.809931 5693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:06.811079 5693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:06.812358 5693 hybrid_clock.cc:648] HybridClock initialized: now 1790681406812284 us; error 71 us; skew 500 ppm
I20260929 11:30:06.812862 5693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:06 dist-test-slave-qk05 krb5kdc[5587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681406, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:06.816980 5693 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:30:06.818804 5693 webserver.cc:460] Webserver started at http://127.23.153.1:40177/ using document root <none> and password file <none>
I20260929 11:30:06.819653 5693 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:06.819788 5693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:06.820109 5693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:30:06.823457 5693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "a82470a638a141f0ade33721fd933851"
format_stamp: "Formatted at 2026-09-29 11:30:06 on dist-test-slave-qk05"
server_key: "7455426161b4ecfe647a966478f54431"
server_key_iv: "59d5d28c7296d97b49608113bd58671f"
server_key_version: "encryptionkey@0"
I20260929 11:30:06.824365 5693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "a82470a638a141f0ade33721fd933851"
format_stamp: "Formatted at 2026-09-29 11:30:06 on dist-test-slave-qk05"
server_key: "7455426161b4ecfe647a966478f54431"
server_key_iv: "59d5d28c7296d97b49608113bd58671f"
server_key_version: "encryptionkey@0"
I20260929 11:30:06.832600 5693 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260929 11:30:06.837209 5708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:06.839999 5693 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 11:30:06.840246 5693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "a82470a638a141f0ade33721fd933851"
format_stamp: "Formatted at 2026-09-29 11:30:06 on dist-test-slave-qk05"
server_key: "7455426161b4ecfe647a966478f54431"
server_key_iv: "59d5d28c7296d97b49608113bd58671f"
server_key_version: "encryptionkey@0"
I20260929 11:30:06.840509 5693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:06.864974 5693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:06.870347 5693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:06.870667 5693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:06.871896 5693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:30:06.873446 5693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:30:06.873543 5693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:06.873677 5693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:30:06.873733 5693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:06.910398 5693 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:44515
I20260929 11:30:06.910478 5820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:44515 every 8 connection(s)
I20260929 11:30:06.911829 5693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:30:06.920974 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 5693
I20260929 11:30:06.921211 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:30:06.922031 24164 external_mini_cluster.cc:1428] Setting key 5e7f684b4b9ec6d44e50bc4e52df6e1b
Sep 29 11:30:06 dist-test-slave-qk05 krb5kdc[5587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681406, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:30:06.939356 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:06.915612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:38639 (local address 127.23.153.62:44055)
0929 11:30:06.916157 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:06.916162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:06.917426 (+ 1264us) server_negotiation.cc:407] Connection header received
0929 11:30:06.920068 (+ 2642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:06.920074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:06.920182 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:06.920284 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:06.922833 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:06.923932 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:06.926770 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:06.927257 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:06.927392 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:06.932267 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:06.932274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:06.932277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:06.932322 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:06.935641 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:06.936604 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:06.936610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:06.936613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:06.936775 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:06.937320 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:06.937325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:06.937327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:06.937679 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:06.937819 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:06.938629 (+ 810us) server_negotiation.cc:299] Negotiation successful
0929 11:30:06.938907 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":278,"threads_started":1}
I20260929 11:30:06.940233 5824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:06.916279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:38639)
0929 11:30:06.917222 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:06.917272 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:06.919677 (+ 2405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:06.920649 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:06.920665 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:06.921531 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:30:06.922526 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:06.922562 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:06.924182 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:06.924188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:06.926482 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:06.926507 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:06.927541 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:06.927548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:06.927879 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:06.928816 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:30:06.928851 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:30:06.931912 (+ 3061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:30:06.935874 (+ 3962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:06.935885 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:06.935902 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:06.936449 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:06.936904 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:06.936908 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:06.936910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:06.937193 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:06.937833 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:30:06.937841 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:30:06.938233 (+ 392us) client_negotiation.cc:769] Sending connection context
0929 11:30:06.938489 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 11:30:06.938966 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":437,"threads_started":1}
I20260929 11:30:06.942322 5821 heartbeater.cc:347] Connected to a master server at 127.23.153.62:44055
I20260929 11:30:06.942741 5821 heartbeater.cc:464] Registering TS with master...
I20260929 11:30:06.943657 5821 heartbeater.cc:511] Master 127.23.153.62:44055 requested a full tablet report, sending...
I20260929 11:30:06.946786 5632 ts_manager.cc:194] Registered new tserver with Master: a82470a638a141f0ade33721fd933851 (127.23.153.1:44515)
WARNING: no policy specified for kudu/127.23.153.2@KRBTEST.COM; defaulting to no policy
I20260929 11:30:06.949093 5632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:38639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
I20260929 11:30:06.982900 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:44055
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:30:07.347880 5827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:07.348232 5827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:07.348376 5827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:07.356493 5827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:07.356647 5827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:07.356802 5827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:30:07.367688 5827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:44055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:07.369125 5827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:07.371256 5827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:07.385289 5832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:07.385326 5833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:07.385258 5835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:07.386114 5827 server_base.cc:1034] running on GCE node
I20260929 11:30:07.386897 5827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:07.388667 5827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:07.389942 5827 hybrid_clock.cc:648] HybridClock initialized: now 1790681407389875 us; error 62 us; skew 500 ppm
I20260929 11:30:07.390328 5827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:07 dist-test-slave-qk05 krb5kdc[5587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681407, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:07.394028 5827 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:30:07.395792 5827 webserver.cc:460] Webserver started at http://127.23.153.2:38607/ using document root <none> and password file <none>
I20260929 11:30:07.396569 5827 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:07.396703 5827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:07.397040 5827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:30:07.400290 5827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "552adac140c443bb9554d1537a14d779"
format_stamp: "Formatted at 2026-09-29 11:30:07 on dist-test-slave-qk05"
server_key: "0b39ec96c4eb8e2c3d185b0350fc6f60"
server_key_iv: "89559dccfa9e7ec6a5aa3c11c129f7cd"
server_key_version: "encryptionkey@0"
I20260929 11:30:07.401144 5827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "552adac140c443bb9554d1537a14d779"
format_stamp: "Formatted at 2026-09-29 11:30:07 on dist-test-slave-qk05"
server_key: "0b39ec96c4eb8e2c3d185b0350fc6f60"
server_key_iv: "89559dccfa9e7ec6a5aa3c11c129f7cd"
server_key_version: "encryptionkey@0"
I20260929 11:30:07.407244 5827 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 11:30:07.411401 5842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:07.413069 5827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:30:07.413270 5827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "552adac140c443bb9554d1537a14d779"
format_stamp: "Formatted at 2026-09-29 11:30:07 on dist-test-slave-qk05"
server_key: "0b39ec96c4eb8e2c3d185b0350fc6f60"
server_key_iv: "89559dccfa9e7ec6a5aa3c11c129f7cd"
server_key_version: "encryptionkey@0"
I20260929 11:30:07.413486 5827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:07.426577 5827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:07.431532 5827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:07.431849 5827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:07.432927 5827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:30:07.434463 5827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:30:07.434545 5827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:07.434687 5827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:30:07.434746 5827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:07.468006 5827 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:42549
I20260929 11:30:07.468060 5954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:42549 every 8 connection(s)
I20260929 11:30:07.469478 5827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:30:07.478673 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 5827
I20260929 11:30:07.478863 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:30:07.479379 24164 external_mini_cluster.cc:1428] Setting key 2113c6bceec1a406173271297ad6454a
Sep 29 11:30:07 dist-test-slave-qk05 krb5kdc[5587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681407, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:30:07.493000 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:07.472243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:55051 (local address 127.23.153.62:44055)
0929 11:30:07.472688 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:07.472693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:07.473605 (+ 912us) server_negotiation.cc:407] Connection header received
0929 11:30:07.475558 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:07.475564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:07.475663 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:07.475761 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:07.477438 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:07.478186 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:07.480304 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:07.481032 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:07.481174 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:07.485910 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:07.485916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:07.485920 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:07.485970 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:07.488934 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:07.490049 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:07.490054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:07.490056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:07.490222 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:07.490790 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:07.490794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:07.490797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:07.491185 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:07.491322 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:07.492294 (+ 972us) server_negotiation.cc:299] Negotiation successful
0929 11:30:07.492641 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":339,"thread_start_us":224,"threads_started":1}
I20260929 11:30:07.494294 5958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:07.472709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:55051)
0929 11:30:07.473444 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:07.473484 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:07.475237 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:07.476046 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:07.476056 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:07.476616 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:30:07.477240 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:07.477260 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:07.478365 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:07.478369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:07.480119 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:07.480144 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:07.481315 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:07.481325 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:07.481617 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:07.482294 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:30:07.482320 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:30:07.485591 (+ 3271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:30:07.489167 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:07.489177 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:07.489195 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:07.489852 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:07.490338 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:07.490343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:07.490347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:07.490633 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:07.491315 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:30:07.491325 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:30:07.491815 (+ 490us) client_negotiation.cc:769] Sending connection context
0929 11:30:07.492142 (+ 327us) client_negotiation.cc:240] Negotiation successful
0929 11:30:07.492753 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":295,"threads_started":1}
I20260929 11:30:07.496224 5955 heartbeater.cc:347] Connected to a master server at 127.23.153.62:44055
I20260929 11:30:07.496608 5955 heartbeater.cc:464] Registering TS with master...
I20260929 11:30:07.497512 5955 heartbeater.cc:511] Master 127.23.153.62:44055 requested a full tablet report, sending...
I20260929 11:30:07.499398 5632 ts_manager.cc:194] Registered new tserver with Master: 552adac140c443bb9554d1537a14d779 (127.23.153.2:42549)
I20260929 11:30:07.500315 5632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:55051
WARNING: no policy specified for kudu/127.23.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
I20260929 11:30:07.535097 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:44055
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:30:07.906445 5961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:07.906765 5961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:07.906863 5961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:07.915122 5961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:07.915251 5961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:07.915378 5961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:30:07.926275 5961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:44055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:07.927723 5961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:07.929117 5961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:07.947142 5969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:07.947252 5966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:07.947180 5967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:07.947986 5961 server_base.cc:1034] running on GCE node
I20260929 11:30:07.948808 5961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:07.950138 5961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:07.951396 5961 hybrid_clock.cc:648] HybridClock initialized: now 1790681407951339 us; error 44 us; skew 500 ppm
I20260929 11:30:07.951826 5961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:30:07.953253 5821 heartbeater.cc:503] Master 127.23.153.62:44055 was elected leader, sending a full tablet report...
Sep 29 11:30:07 dist-test-slave-qk05 krb5kdc[5587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681407, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:07.955757 5961 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:30:07.957485 5961 webserver.cc:460] Webserver started at http://127.23.153.3:37449/ using document root <none> and password file <none>
I20260929 11:30:07.958257 5961 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:07.958377 5961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:07.958668 5961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:30:07.962234 5961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "4c7d1b658b4242f18129bded75a929df"
format_stamp: "Formatted at 2026-09-29 11:30:07 on dist-test-slave-qk05"
server_key: "6e785a7ab3d929a4c6294ecfddc83a76"
server_key_iv: "9365431a44faefd3b7e73dbb2e95346e"
server_key_version: "encryptionkey@0"
I20260929 11:30:07.963205 5961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "4c7d1b658b4242f18129bded75a929df"
format_stamp: "Formatted at 2026-09-29 11:30:07 on dist-test-slave-qk05"
server_key: "6e785a7ab3d929a4c6294ecfddc83a76"
server_key_iv: "9365431a44faefd3b7e73dbb2e95346e"
server_key_version: "encryptionkey@0"
I20260929 11:30:07.969055 5961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 11:30:07.973140 5976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:07.974872 5961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:30:07.975107 5961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "4c7d1b658b4242f18129bded75a929df"
format_stamp: "Formatted at 2026-09-29 11:30:07 on dist-test-slave-qk05"
server_key: "6e785a7ab3d929a4c6294ecfddc83a76"
server_key_iv: "9365431a44faefd3b7e73dbb2e95346e"
server_key_version: "encryptionkey@0"
I20260929 11:30:07.975325 5961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:07.997063 5961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:08.001799 5961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:08.002066 5961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:08.003154 5961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:30:08.004791 5961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:30:08.004884 5961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:08.004972 5961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:30:08.005015 5961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:08.037618 5961 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:40885
I20260929 11:30:08.037673 6088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:40885 every 8 connection(s)
I20260929 11:30:08.039005 5961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:30:08.042168 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 5961
I20260929 11:30:08.042363 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:30:08.042923 24164 external_mini_cluster.cc:1428] Setting key 4452705099f3038eec0364e5f7e2105c
Sep 29 11:30:08 dist-test-slave-qk05 krb5kdc[5587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681407, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:30:08.065160 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:08.041853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:38851 (local address 127.23.153.62:44055)
0929 11:30:08.042338 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:08.042345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:08.043737 (+ 1392us) server_negotiation.cc:407] Connection header received
0929 11:30:08.046028 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:08.046041 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:08.046153 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:08.046277 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:08.048121 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.048998 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.050815 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.051293 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.051409 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:08.055997 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:08.056003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:08.056007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:08.056066 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:08.060040 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:08.061409 (+ 1369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:08.061416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:08.061419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:08.061680 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:08.062427 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:08.062431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:08.062433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:08.062831 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:08.063003 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:08.064331 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0929 11:30:08.064723 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":247,"threads_started":1}
I20260929 11:30:08.066256 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:08.042579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:38851)
0929 11:30:08.043518 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:08.043575 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:08.045727 (+ 2152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:08.046532 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:08.046542 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:08.047208 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:30:08.047891 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:08.047911 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.049199 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.049203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:08.050617 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:08.050634 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.051558 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.051564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:08.051793 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:08.052516 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:30:08.052542 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:30:08.055667 (+ 3125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:30:08.060226 (+ 4559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:08.060236 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:08.060252 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:08.061139 (+ 887us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:08.061932 (+ 793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:08.061938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:08.061942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:08.062264 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:08.062992 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:30:08.063003 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:30:08.063576 (+ 573us) client_negotiation.cc:769] Sending connection context
0929 11:30:08.064134 (+ 558us) client_negotiation.cc:240] Negotiation successful
0929 11:30:08.064953 (+ 819us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":407,"threads_started":1}
I20260929 11:30:08.067986 6089 heartbeater.cc:347] Connected to a master server at 127.23.153.62:44055
I20260929 11:30:08.068324 6089 heartbeater.cc:464] Registering TS with master...
I20260929 11:30:08.069125 6089 heartbeater.cc:511] Master 127.23.153.62:44055 requested a full tablet report, sending...
I20260929 11:30:08.070940 5632 ts_manager.cc:194] Registered new tserver with Master: 4c7d1b658b4242f18129bded75a929df (127.23.153.3:40885)
I20260929 11:30:08.071960 5632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:38851
I20260929 11:30:08.072377 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:30:08.091159 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:08.077042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45930 (local address 127.23.153.62:44055)
0929 11:30:08.077287 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:08.077294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:08.077706 (+ 412us) server_negotiation.cc:407] Connection header received
0929 11:30:08.077783 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:08.077786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:08.077895 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:08.077995 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:08.079551 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.080780 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.082228 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.082676 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.082791 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:08.085444 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:08.085451 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:08.085454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:08.085496 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:08.088232 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:08.089105 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:08.089110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:08.089112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:08.089268 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:08.089674 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:08.089678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:08.089680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:08.090052 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:08.090258 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:08.090588 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 11:30:08.090797 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 11:30:08.096031 5632 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:30:08.104802 5632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:30:08.147265 6107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:08.122499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53380 (local address 127.23.153.3:40885)
0929 11:30:08.136617 (+ 14118us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:08.136628 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:08.136671 (+ 43us) server_negotiation.cc:407] Connection header received
0929 11:30:08.136807 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:08.136817 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:08.137149 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:08.137431 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:08.139434 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.141000 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.144223 (+ 3223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.145809 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.145952 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:08.146255 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:08.146627 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 11:30:08.146874 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13870,"thread_start_us":15273,"threads_started":1}
I20260929 11:30:08.147294 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:08.122293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:40885 (local address 127.0.0.1:53380)
0929 11:30:08.135779 (+ 13486us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:08.135800 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:08.135943 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:08.138163 (+ 2220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:08.138173 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:08.138223 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:08.138720 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:08.138735 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.141193 (+ 2458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.141198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:08.143967 (+ 2769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:08.143989 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.146051 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.146058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:08.146316 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:08.146375 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 11:30:08.146574 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 11:30:08.146816 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13350,"mutex_wait_us":58,"thread_start_us":13196,"threads_started":1}
I20260929 11:30:08.150485 6103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:08.121101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:42549 (local address 127.0.0.1:60712)
0929 11:30:08.128354 (+ 7253us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:08.128425 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:08.128683 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:08.140338 (+ 11655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:08.140343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:08.140375 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:08.140809 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:08.140822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.142809 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.142816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:08.145740 (+ 2924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:08.145770 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.149412 (+ 3642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.149420 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:08.149724 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:08.149766 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 11:30:08.149885 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 11:30:08.150047 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6864,"mutex_wait_us":144,"thread_start_us":14109,"threads_started":1}
I20260929 11:30:08.150535 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:08.121849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.23.153.2:42549)
0929 11:30:08.139467 (+ 17618us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:08.139479 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:08.139517 (+ 38us) server_negotiation.cc:407] Connection header received
0929 11:30:08.139744 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:08.139757 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:08.140074 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:08.140324 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:08.141022 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.142606 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.146346 (+ 3740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.149133 (+ 2787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.149340 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:08.149567 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:08.149975 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 11:30:08.150189 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17391,"thread_start_us":17211,"threads_started":1}
I20260929 11:30:08.151634 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:08.121722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.23.153.1:44515)
0929 11:30:08.140076 (+ 18354us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:08.140084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:08.140113 (+ 29us) server_negotiation.cc:407] Connection header received
0929 11:30:08.140208 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:08.140214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:08.140427 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:08.140949 (+ 522us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:08.141887 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.143339 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.147063 (+ 3724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.149788 (+ 2725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.149940 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:08.150161 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:08.151139 (+ 978us) server_negotiation.cc:299] Negotiation successful
0929 11:30:08.151332 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18174,"thread_start_us":18957,"threads_started":1}
I20260929 11:30:08.151623 6102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:08.121450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:44515 (local address 127.0.0.1:44006)
0929 11:30:08.135585 (+ 14135us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:08.135610 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:08.135747 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:08.140964 (+ 5217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:08.140969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:08.140998 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:08.141474 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:08.141526 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.143524 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.143530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:08.146658 (+ 3128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:08.146678 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.150321 (+ 3643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.150332 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:08.150538 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:08.150897 (+ 359us) client_negotiation.cc:769] Sending connection context
0929 11:30:08.151035 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 11:30:08.151197 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13972,"mutex_wait_us":41,"thread_start_us":6331,"threads_started":1}
I20260929 11:30:08.152777 6024 tablet_service.cc:1467] Processing CreateTablet for tablet 1066fa6f31244138a292324f4396888c (DEFAULT_TABLE table=test-table [id=d849b4d408c2413c85e7aac19921798b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:30:08.155520 6024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1066fa6f31244138a292324f4396888c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:30:08.155282 5890 tablet_service.cc:1467] Processing CreateTablet for tablet 1066fa6f31244138a292324f4396888c (DEFAULT_TABLE table=test-table [id=d849b4d408c2413c85e7aac19921798b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:30:08.155797 5756 tablet_service.cc:1467] Processing CreateTablet for tablet 1066fa6f31244138a292324f4396888c (DEFAULT_TABLE table=test-table [id=d849b4d408c2413c85e7aac19921798b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:30:08.156579 5890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1066fa6f31244138a292324f4396888c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:30:08.157246 5756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1066fa6f31244138a292324f4396888c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:30:08.168697 6108 tablet_bootstrap.cc:492] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779: Bootstrap starting.
I20260929 11:30:08.172351 6108 tablet_bootstrap.cc:654] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779: Neither blocks nor log segments found. Creating new log.
I20260929 11:30:08.173353 6109 tablet_bootstrap.cc:492] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851: Bootstrap starting.
I20260929 11:30:08.174288 6108 log.cc:824] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:08.175585 6110 tablet_bootstrap.cc:492] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df: Bootstrap starting.
I20260929 11:30:08.177752 6109 tablet_bootstrap.cc:654] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851: Neither blocks nor log segments found. Creating new log.
I20260929 11:30:08.178913 6110 tablet_bootstrap.cc:654] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df: Neither blocks nor log segments found. Creating new log.
I20260929 11:30:08.179476 6109 log.cc:824] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:08.180078 6110 log.cc:824] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:08.184585 6110 tablet_bootstrap.cc:492] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df: No bootstrap required, opened a new log
I20260929 11:30:08.184757 6108 tablet_bootstrap.cc:492] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779: No bootstrap required, opened a new log
I20260929 11:30:08.184931 6110 ts_tablet_manager.cc:1397] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260929 11:30:08.185091 6108 ts_tablet_manager.cc:1397] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.005s
I20260929 11:30:08.190826 6109 tablet_bootstrap.cc:492] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851: No bootstrap required, opened a new log
I20260929 11:30:08.191246 6109 ts_tablet_manager.cc:1397] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.011s
I20260929 11:30:08.193178 6110 raft_consensus.cc:348] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } } peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } }
I20260929 11:30:08.193620 6110 raft_consensus.cc:374] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:30:08.193722 6110 raft_consensus.cc:729] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c7d1b658b4242f18129bded75a929df, State: Initialized, Role: FOLLOWER
I20260929 11:30:08.194599 6110 consensus_queue.cc:260] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } } peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } }
I20260929 11:30:08.196527 6089 heartbeater.cc:503] Master 127.23.153.62:44055 was elected leader, sending a full tablet report...
I20260929 11:30:08.196244 6108 raft_consensus.cc:348] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } } peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } }
I20260929 11:30:08.196800 6108 raft_consensus.cc:374] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:30:08.196904 6108 raft_consensus.cc:729] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 552adac140c443bb9554d1537a14d779, State: Initialized, Role: FOLLOWER
I20260929 11:30:08.197726 6108 consensus_queue.cc:260] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } } peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } }
I20260929 11:30:08.198813 6110 ts_tablet_manager.cc:1428] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 11:30:08.203658 6109 raft_consensus.cc:348] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } } peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } }
I20260929 11:30:08.204226 6109 raft_consensus.cc:374] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:30:08.204325 6109 raft_consensus.cc:729] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a82470a638a141f0ade33721fd933851, State: Initialized, Role: FOLLOWER
I20260929 11:30:08.205056 6109 consensus_queue.cc:260] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } } peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } }
I20260929 11:30:08.206373 5955 heartbeater.cc:503] Master 127.23.153.62:44055 was elected leader, sending a full tablet report...
I20260929 11:30:08.207573 6108 ts_tablet_manager.cc:1428] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779: Time spent starting tablet: real 0.022s user 0.014s sys 0.008s
I20260929 11:30:08.219004 6115 raft_consensus.cc:482] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:30:08.219484 6115 raft_consensus.cc:504] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } } peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } }
I20260929 11:30:08.219928 6109 ts_tablet_manager.cc:1428] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851: Time spent starting tablet: real 0.029s user 0.021s sys 0.006s
W20260929 11:30:08.221524 5956 tablet.cc:2369] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:30:08.226475 6115 leader_election.cc:288] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a82470a638a141f0ade33721fd933851 (127.23.153.1:44515), 4c7d1b658b4242f18129bded75a929df (127.23.153.3:40885)
I20260929 11:30:08.233665 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:08.223713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:59009 (local address 127.23.153.1:44515)
0929 11:30:08.225184 (+ 1471us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:08.225190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:08.226675 (+ 1485us) server_negotiation.cc:407] Connection header received
0929 11:30:08.227420 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:08.227425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:08.227494 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:08.227570 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:08.228218 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.229319 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.231605 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.232521 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.232636 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:08.232695 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:08.233267 (+ 572us) server_negotiation.cc:299] Negotiation successful
0929 11:30:08.233416 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1309}
I20260929 11:30:08.233661 6107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:08.223356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:53489 (local address 127.23.153.3:40885)
0929 11:30:08.223696 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:08.223703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:08.227821 (+ 4118us) server_negotiation.cc:407] Connection header received
0929 11:30:08.228125 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:08.228131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:08.228214 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:08.228292 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:08.228824 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.229581 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.231776 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.232631 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.232791 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:08.232866 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:08.233292 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 11:30:08.233424 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260929 11:30:08.233829 6117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:08.223165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:44515 (local address 127.23.153.2:59009)
0929 11:30:08.226472 (+ 3307us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:08.226494 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:08.227230 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:08.227618 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:08.227623 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:08.227651 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:08.228014 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:08.228024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.229466 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.229472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:08.231380 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:08.231396 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.232884 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.232888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:08.233080 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:08.233121 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 11:30:08.233198 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 11:30:08.233345 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3156,"mutex_wait_us":46,"thread_start_us":2965,"threads_started":1}
I20260929 11:30:08.233855 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:08.223194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:40885 (local address 127.23.153.2:53489)
0929 11:30:08.227525 (+ 4331us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:08.227549 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:08.227962 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:08.228321 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:08.228325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:08.228350 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:08.228674 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:08.228683 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.229729 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.229733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:08.231525 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:08.231544 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.232881 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.232889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:08.233120 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:08.233146 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 11:30:08.233236 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 11:30:08.233375 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4149,"mutex_wait_us":39,"spinlock_wait_cycles":28416,"thread_start_us":3748,"threads_started":1}
I20260929 11:30:08.234854 5776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1066fa6f31244138a292324f4396888c" candidate_uuid: "552adac140c443bb9554d1537a14d779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a82470a638a141f0ade33721fd933851" is_pre_election: true
I20260929 11:30:08.235425 5776 raft_consensus.cc:2454] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 552adac140c443bb9554d1537a14d779 in term 0.
I20260929 11:30:08.235340 6044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1066fa6f31244138a292324f4396888c" candidate_uuid: "552adac140c443bb9554d1537a14d779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c7d1b658b4242f18129bded75a929df" is_pre_election: true
I20260929 11:30:08.235787 6044 raft_consensus.cc:2454] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 552adac140c443bb9554d1537a14d779 in term 0.
I20260929 11:30:08.236167 5846 leader_election.cc:302] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 552adac140c443bb9554d1537a14d779, a82470a638a141f0ade33721fd933851; no voters:
I20260929 11:30:08.236606 6115 raft_consensus.cc:2781] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:30:08.236765 6115 raft_consensus.cc:482] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:30:08.236868 6115 raft_consensus.cc:3037] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:30:08.239180 6115 raft_consensus.cc:504] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } } peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } }
I20260929 11:30:08.239914 6115 leader_election.cc:288] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [CANDIDATE]: Term 1 election: Requested vote from peers a82470a638a141f0ade33721fd933851 (127.23.153.1:44515), 4c7d1b658b4242f18129bded75a929df (127.23.153.3:40885)
I20260929 11:30:08.240624 5776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1066fa6f31244138a292324f4396888c" candidate_uuid: "552adac140c443bb9554d1537a14d779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a82470a638a141f0ade33721fd933851"
I20260929 11:30:08.240613 6044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1066fa6f31244138a292324f4396888c" candidate_uuid: "552adac140c443bb9554d1537a14d779" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c7d1b658b4242f18129bded75a929df"
I20260929 11:30:08.240866 5776 raft_consensus.cc:3037] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:30:08.240877 6044 raft_consensus.cc:3037] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:30:08.244288 6044 raft_consensus.cc:2454] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 552adac140c443bb9554d1537a14d779 in term 1.
I20260929 11:30:08.244288 5776 raft_consensus.cc:2454] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 552adac140c443bb9554d1537a14d779 in term 1.
I20260929 11:30:08.244940 5846 leader_election.cc:302] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4c7d1b658b4242f18129bded75a929df, 552adac140c443bb9554d1537a14d779, a82470a638a141f0ade33721fd933851; no voters:
I20260929 11:30:08.245375 6115 raft_consensus.cc:2781] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:30:08.245908 6115 raft_consensus.cc:686] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [term 1 LEADER]: Becoming Leader. State: Replica: 552adac140c443bb9554d1537a14d779, State: Running, Role: LEADER
I20260929 11:30:08.246479 6115 consensus_queue.cc:237] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } } peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } }
I20260929 11:30:08.251495 5632 catalog_manager.cc:5495] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 reported cstate change: term changed from 0 to 1, leader changed from <none> to 552adac140c443bb9554d1537a14d779 (127.23.153.2). New cstate: current_term: 1 leader_uuid: "552adac140c443bb9554d1537a14d779" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } health_report { overall_health: UNKNOWN } } }
I20260929 11:30:08.270119 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:08.264377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60718 (local address 127.23.153.2:42549)
0929 11:30:08.264609 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:08.264615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:08.264658 (+ 43us) server_negotiation.cc:407] Connection header received
0929 11:30:08.264929 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:08.264933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:08.265027 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:08.265171 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:30:08.265587 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.266333 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.267860 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.268278 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.268397 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:08.268928 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:30:08.269475 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:30:08.269626 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:08.269738 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 11:30:08.269832 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 11:30:08.279853 5776 raft_consensus.cc:1261] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851 [term 1 FOLLOWER]: Refusing update from remote peer 552adac140c443bb9554d1537a14d779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:30:08.279853 6044 raft_consensus.cc:1261] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df [term 1 FOLLOWER]: Refusing update from remote peer 552adac140c443bb9554d1537a14d779: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:30:08.281127 6119 consensus_queue.cc:1035] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:30:08.281546 6115 consensus_queue.cc:1035] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 11:30:08.290817 6090 tablet.cc:2369] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:30:08.299520 6122 mvcc.cc:205] Tried to move back new op lower bound from 7334631048300744704 to 7334631048181923840. Current Snapshot: MvccSnapshot[applied={T|T < 7334631048300744704}]
I20260929 11:30:08.300611 6124 mvcc.cc:205] Tried to move back new op lower bound from 7334631048300744704 to 7334631048181923840. Current Snapshot: MvccSnapshot[applied={T|T < 7334631048300744704}]
I20260929 11:30:08.322417 5632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:30:08.322701 5632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:30:08.326960 5632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402: Sending DeleteTablet for 3 replicas of tablet 1066fa6f31244138a292324f4396888c
I20260929 11:30:08.328344 5890 tablet_service.cc:1514] Processing DeleteTablet for tablet 1066fa6f31244138a292324f4396888c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:30:08 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:60712
I20260929 11:30:08.328341 6024 tablet_service.cc:1514] Processing DeleteTablet for tablet 1066fa6f31244138a292324f4396888c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:30:08 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:53380
I20260929 11:30:08.328341 5756 tablet_service.cc:1514] Processing DeleteTablet for tablet 1066fa6f31244138a292324f4396888c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:30:08 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:44006
I20260929 11:30:08.330097 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 5693
I20260929 11:30:08.340814 6133 tablet_replica.cc:330] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779: stopping tablet replica
I20260929 11:30:08.340869 6134 tablet_replica.cc:330] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851: stopping tablet replica
I20260929 11:30:08.341497 6134 raft_consensus.cc:2229] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:30:08.341534 6133 raft_consensus.cc:2229] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:30:08.342155 6134 raft_consensus.cc:2258] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:30:08.342582 6133 raft_consensus.cc:2258] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:30:08.343336 6135 tablet_replica.cc:330] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df: stopping tablet replica
I20260929 11:30:08.343814 6135 raft_consensus.cc:2229] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:30:08.344349 6135 raft_consensus.cc:2258] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:30:08.345971 6133 ts_tablet_manager.cc:1905] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:30:08.346120 6134 ts_tablet_manager.cc:1905] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:30:08.350798 6135 ts_tablet_manager.cc:1905] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 11:30:08 dist-test-slave-qk05 krb5kdc[5587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:30:08.354755 6133 ts_tablet_manager.cc:1918] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:30:08.355129 6133 log.cc:1196] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/1066fa6f31244138a292324f4396888c
I20260929 11:30:08.355875 6133 ts_tablet_manager.cc:1939] T 1066fa6f31244138a292324f4396888c P 552adac140c443bb9554d1537a14d779: Deleting consensus metadata
I20260929 11:30:08.358280 5617 catalog_manager.cc:4841] TS 552adac140c443bb9554d1537a14d779 (127.23.153.2:42549): tablet 1066fa6f31244138a292324f4396888c (table test-table [id=d849b4d408c2413c85e7aac19921798b]) successfully deleted
I20260929 11:30:08.360423 6134 ts_tablet_manager.cc:1918] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:30:08.360735 6134 log.cc:1196] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/1066fa6f31244138a292324f4396888c
I20260929 11:30:08.361347 6134 ts_tablet_manager.cc:1939] T 1066fa6f31244138a292324f4396888c P a82470a638a141f0ade33721fd933851: Deleting consensus metadata
I20260929 11:30:08.361857 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:08.331500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44018 (local address 127.23.153.1:44515)
0929 11:30:08.331703 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:08.331710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:08.339618 (+ 7908us) server_negotiation.cc:407] Connection header received
0929 11:30:08.339963 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:08.339971 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:08.340097 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:08.340179 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:08.342444 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.343593 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.345981 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.346667 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.346792 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:08.352750 (+ 5958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:08.352774 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:08.352798 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:08.352854 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:08.357156 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:08.358460 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:08.358471 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:08.358474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:08.358831 (+ 357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:08.359460 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:08.359465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:08.359468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:08.360002 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:08.360325 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:08.361157 (+ 832us) server_negotiation.cc:299] Negotiation successful
0929 11:30:08.361452 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 11:30:08.363942 5816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:08.364063 5619 catalog_manager.cc:4841] TS a82470a638a141f0ade33721fd933851 (127.23.153.1:44515): tablet 1066fa6f31244138a292324f4396888c (table test-table [id=d849b4d408c2413c85e7aac19921798b]) successfully deleted
I20260929 11:30:08.365255 6135 ts_tablet_manager.cc:1918] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:30:08.366155 6135 log.cc:1196] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/1066fa6f31244138a292324f4396888c
I20260929 11:30:08.367991 6135 ts_tablet_manager.cc:1939] T 1066fa6f31244138a292324f4396888c P 4c7d1b658b4242f18129bded75a929df: Deleting consensus metadata
I20260929 11:30:08.370707 5618 catalog_manager.cc:4841] TS 4c7d1b658b4242f18129bded75a929df (127.23.153.3:40885): tablet 1066fa6f31244138a292324f4396888c (table test-table [id=d849b4d408c2413c85e7aac19921798b]) successfully deleted
I20260929 11:30:08.480825 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 5693
I20260929 11:30:08.503309 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 5827
Sep 29 11:30:08 dist-test-slave-qk05 krb5kdc[5587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.2@KRBTEST.COM
I20260929 11:30:08.520982 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:08.504342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60734 (local address 127.23.153.2:42549)
0929 11:30:08.504519 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:08.504523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:08.504805 (+ 282us) server_negotiation.cc:407] Connection header received
0929 11:30:08.504994 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:08.504999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:08.505075 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:08.505163 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:08.506537 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.507574 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.509010 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.509486 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.509627 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:08.514278 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:08.514298 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:08.514316 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:08.514360 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:08.517696 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:08.518635 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:08.518644 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:08.518647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:08.518802 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:08.519272 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:08.519277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:08.519279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:08.519674 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:08.520050 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:08.520331 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 11:30:08.520609 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:30:08.522047 5950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:08.638897 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 5827
I20260929 11:30:08.658036 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 5961
Sep 29 11:30:08 dist-test-slave-qk05 krb5kdc[5587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.3@KRBTEST.COM
I20260929 11:30:08.678678 6107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:08.659121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53394 (local address 127.23.153.3:40885)
0929 11:30:08.659354 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:08.659362 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:08.659460 (+ 98us) server_negotiation.cc:407] Connection header received
0929 11:30:08.659881 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:08.659887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:08.659971 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:08.660084 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:08.662260 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.663218 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.665320 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:08.665784 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:08.665910 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:08.671241 (+ 5331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:08.671260 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:08.671279 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:08.671335 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:08.674827 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:08.676063 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:08.676072 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:08.676075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:08.676276 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:08.676923 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:08.676928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:08.676930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:08.677394 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:08.677677 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:08.678149 (+ 472us) server_negotiation.cc:299] Negotiation successful
0929 11:30:08.678363 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 11:30:08.679767 6084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:08.773531 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 5961
I20260929 11:30:08.788414 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 5600
I20260929 11:30:08.789729 5662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:08.903316 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 5600
I20260929 11:30:08.923117 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:44055
--webserver_interface=127.23.153.62
--webserver_port=39609
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:44055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:30:09.291981 6141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:09.292341 6141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:09.292462 6141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:09.300879 6141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:30:09.301019 6141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:09.301076 6141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:09.301162 6141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:30:09.301213 6141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:30:09.312988 6141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:44055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:44055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=39609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:09.314471 6141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:09.316025 6141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:09.340155 6147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:09.340188 6150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:09.340133 6148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:09.341063 6141 server_base.cc:1034] running on GCE node
I20260929 11:30:09.342052 6141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:09.344002 6141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:09.345294 6141 hybrid_clock.cc:648] HybridClock initialized: now 1790681409345227 us; error 79 us; skew 500 ppm
I20260929 11:30:09.345783 6141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:09 dist-test-slave-qk05 krb5kdc[5587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681409, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:09.350107 6141 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:30:09.352165 6141 webserver.cc:460] Webserver started at http://127.23.153.62:39609/ using document root <none> and password file <none>
I20260929 11:30:09.353016 6141 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:09.353190 6141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:09.360179 6141 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 11:30:09.364220 6157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:09.366250 6141 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:30:09.366500 6141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "076d6020653b415c9439019df315f402"
format_stamp: "Formatted at 2026-09-29 11:30:06 on dist-test-slave-qk05"
server_key: "03ac5bb2b8d499ca591bec6422281109"
server_key_iv: "b76f1e61989ea99354821039a1af583e"
server_key_version: "encryptionkey@0"
I20260929 11:30:09.367807 6141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:09.407995 6141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:09.412808 6141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:09.413094 6141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:09.432490 6141 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:44055
I20260929 11:30:09.432510 6208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:44055 every 8 connection(s)
I20260929 11:30:09.434074 6141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:30:09.439137 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 6141
I20260929 11:30:09.440091 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:44515
--local_ip_for_outbound_sockets=127.23.153.1
--tserver_master_addrs=127.23.153.62:44055
--webserver_port=40177
--webserver_interface=127.23.153.1
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:30:09.443806 6209 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:30:09.449235 6209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402: Bootstrap starting.
I20260929 11:30:09.472771 6209 log.cc:824] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:09.486094 6209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:30:09.486696 6209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402: Bootstrap complete.
I20260929 11:30:09.496254 6209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076d6020653b415c9439019df315f402" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44055 } }
I20260929 11:30:09.497210 6209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 076d6020653b415c9439019df315f402, State: Initialized, Role: FOLLOWER
I20260929 11:30:09.497896 6209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076d6020653b415c9439019df315f402" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44055 } }
I20260929 11:30:09.498174 6209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:30:09.498310 6209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:30:09.498507 6209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 1 FOLLOWER]: Advancing to term 2
I20260929 11:30:09.502815 6209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076d6020653b415c9439019df315f402" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44055 } }
I20260929 11:30:09.503464 6209 leader_election.cc:302] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 076d6020653b415c9439019df315f402; no voters:
I20260929 11:30:09.504101 6209 leader_election.cc:288] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 11:30:09.504245 6213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 11:30:09.505502 6213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 2 LEADER]: Becoming Leader. State: Replica: 076d6020653b415c9439019df315f402, State: Running, Role: LEADER
I20260929 11:30:09.506120 6213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076d6020653b415c9439019df315f402" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44055 } }
I20260929 11:30:09.506572 6209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:30:09.508867 6215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 076d6020653b415c9439019df315f402. Latest consensus state: current_term: 2 leader_uuid: "076d6020653b415c9439019df315f402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076d6020653b415c9439019df315f402" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44055 } } }
I20260929 11:30:09.509269 6215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [sys.catalog]: This master's current role is: LEADER
I20260929 11:30:09.509879 6214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "076d6020653b415c9439019df315f402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076d6020653b415c9439019df315f402" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44055 } } }
I20260929 11:30:09.510195 6214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [sys.catalog]: This master's current role is: LEADER
I20260929 11:30:09.516278 6220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:30:09.525359 6220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:30:09.527415 6220 catalog_manager.cc:1259] Loaded cluster ID: 53863fcc9f3c46b2bb4eb3860d0f5679
I20260929 11:30:09.527561 6220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:30:09.531105 6220 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 11:30:09.539640 6230 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 11:30:09.549906 6220 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 11:30:09.560374 6220 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:30:09.562878 6220 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402: Loaded TSK: 0
I20260929 11:30:09.564805 6220 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 11:30:09.833205 6211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:09.833549 6211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:09.833696 6211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:09.841902 6211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:09.842065 6211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:09.842217 6211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:30:09.854128 6211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:44515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=40177
--tserver_master_addrs=127.23.153.62:44055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:09.855618 6211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:09.857074 6211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:09.874181 6237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:09.874176 6236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:09.875087 6239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:09.875428 6211 server_base.cc:1034] running on GCE node
I20260929 11:30:09.876137 6211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:09.877240 6211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:09.878472 6211 hybrid_clock.cc:648] HybridClock initialized: now 1790681409878410 us; error 61 us; skew 500 ppm
I20260929 11:30:09.878831 6211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:09 dist-test-slave-qk05 krb5kdc[5587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681409, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:09.883083 6211 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:30:09.885218 6211 webserver.cc:460] Webserver started at http://127.23.153.1:40177/ using document root <none> and password file <none>
I20260929 11:30:09.886281 6211 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:09.886467 6211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:09.894886 6211 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 11:30:09.898743 6246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:09.900761 6211 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260929 11:30:09.901008 6211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "a82470a638a141f0ade33721fd933851"
format_stamp: "Formatted at 2026-09-29 11:30:06 on dist-test-slave-qk05"
server_key: "7455426161b4ecfe647a966478f54431"
server_key_iv: "59d5d28c7296d97b49608113bd58671f"
server_key_version: "encryptionkey@0"
I20260929 11:30:09.902523 6211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:09.935380 6211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:09.941398 6211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:09.941725 6211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:09.943092 6211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:30:09.944890 6211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:30:09.945043 6211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:09.945151 6211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:30:09.945221 6211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:09.978209 6211 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:44515
I20260929 11:30:09.978271 6358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:44515 every 8 connection(s)
I20260929 11:30:09.979854 6211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:30:09.989672 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 6211
I20260929 11:30:09.990479 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:42549
--local_ip_for_outbound_sockets=127.23.153.2
--tserver_master_addrs=127.23.153.62:44055
--webserver_port=38607
--webserver_interface=127.23.153.2
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:30:09 dist-test-slave-qk05 krb5kdc[5587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681409, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:30:10.008133 6362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:09.983309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:39013)
0929 11:30:09.984311 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:09.984366 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:09.986190 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:09.988044 (+ 1854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:09.988054 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:09.988748 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:30:09.989398 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:09.989420 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:09.992130 (+ 2710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:09.992135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:09.993574 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:09.993595 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:09.995229 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:09.995236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:09.995457 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:09.996176 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:30:09.996201 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:30:09.998706 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:30:10.002817 (+ 4111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:10.002826 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:10.002839 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:10.003438 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:10.004000 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:10.004004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:10.004007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:10.004216 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:10.005065 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:30:10.005075 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:30:10.005442 (+ 367us) client_negotiation.cc:769] Sending connection context
0929 11:30:10.005726 (+ 284us) client_negotiation.cc:240] Negotiation successful
0929 11:30:10.006644 (+ 918us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":456,"threads_started":1}
I20260929 11:30:10.008162 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:09.983119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:39013 (local address 127.23.153.62:44055)
0929 11:30:09.983699 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:09.983713 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:09.984521 (+ 808us) server_negotiation.cc:407] Connection header received
0929 11:30:09.986794 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:09.986809 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:09.987444 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:09.987859 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:09.989774 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:09.991855 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:09.993793 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:09.994915 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:09.995187 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:09.999111 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:09.999122 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:09.999140 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:09.999187 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:10.002572 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:10.003627 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:10.003635 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:10.003642 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:10.003851 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:10.004340 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:10.004344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:10.004346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:10.004893 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:10.005122 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:10.006061 (+ 939us) server_negotiation.cc:299] Negotiation successful
0929 11:30:10.006747 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":248,"threads_started":1}
I20260929 11:30:10.012248 6359 heartbeater.cc:347] Connected to a master server at 127.23.153.62:44055
I20260929 11:30:10.012593 6359 heartbeater.cc:464] Registering TS with master...
I20260929 11:30:10.013413 6359 heartbeater.cc:511] Master 127.23.153.62:44055 requested a full tablet report, sending...
I20260929 11:30:10.016113 6174 ts_manager.cc:194] Registered new tserver with Master: a82470a638a141f0ade33721fd933851 (127.23.153.1:44515)
I20260929 11:30:10.017691 6174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:39013
W20260929 11:30:10.351056 6363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:10.351387 6363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:10.351528 6363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:10.359205 6363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:10.359354 6363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:10.359517 6363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:30:10.369817 6363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:42549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=38607
--tserver_master_addrs=127.23.153.62:44055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:10.371233 6363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:10.372654 6363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:10.386843 6371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:10.387006 6369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:10.387056 6368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:10.393194 6363 server_base.cc:1034] running on GCE node
I20260929 11:30:10.394032 6363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:10.395287 6363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:10.396538 6363 hybrid_clock.cc:648] HybridClock initialized: now 1790681410396478 us; error 58 us; skew 500 ppm
I20260929 11:30:10.396915 6363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:10 dist-test-slave-qk05 krb5kdc[5587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681410, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:10.401379 6363 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:30:10.403949 6363 webserver.cc:460] Webserver started at http://127.23.153.2:38607/ using document root <none> and password file <none>
I20260929 11:30:10.405092 6363 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:10.405263 6363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:10.412273 6363 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:30:10.415408 6378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:10.417165 6363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:30:10.417344 6363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "552adac140c443bb9554d1537a14d779"
format_stamp: "Formatted at 2026-09-29 11:30:07 on dist-test-slave-qk05"
server_key: "0b39ec96c4eb8e2c3d185b0350fc6f60"
server_key_iv: "89559dccfa9e7ec6a5aa3c11c129f7cd"
server_key_version: "encryptionkey@0"
I20260929 11:30:10.418406 6363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:10.446694 6363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:10.453146 6363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:10.453455 6363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:10.454727 6363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:30:10.456415 6363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:30:10.456519 6363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:10.456620 6363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:30:10.456681 6363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:10.492635 6363 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:42549
I20260929 11:30:10.492729 6490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:42549 every 8 connection(s)
I20260929 11:30:10.494202 6363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:30:10.497046 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 6363
I20260929 11:30:10.497711 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:40885
--local_ip_for_outbound_sockets=127.23.153.3
--tserver_master_addrs=127.23.153.62:44055
--webserver_port=37449
--webserver_interface=127.23.153.3
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:30:10 dist-test-slave-qk05 krb5kdc[5587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681410, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:30:10.520885 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:10.497242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:53557 (local address 127.23.153.62:44055)
0929 11:30:10.497462 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:10.497468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:10.499682 (+ 2214us) server_negotiation.cc:407] Connection header received
0929 11:30:10.502347 (+ 2665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:10.502355 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:10.502660 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:10.502902 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:10.505593 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:10.506929 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:10.509198 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:10.509782 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:10.509935 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:10.514409 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:10.514414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:10.514418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:10.514459 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:10.517301 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:10.518218 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:10.518223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:10.518226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:10.518384 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:10.518867 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:10.518872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:10.518874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:10.519238 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:10.519390 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:10.520247 (+ 857us) server_negotiation.cc:299] Negotiation successful
0929 11:30:10.520523 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 11:30:10.522464 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:10.497878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:53557)
0929 11:30:10.499158 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:10.499329 (+ 171us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:10.501832 (+ 2503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:10.503327 (+ 1495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:10.503340 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:10.504290 (+ 950us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:30:10.505239 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:10.505271 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:10.507297 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:10.507304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:10.508881 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:10.508907 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:10.510096 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:10.510106 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:10.510360 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:10.511216 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:30:10.511244 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:30:10.514104 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:30:10.517507 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:10.517519 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:10.517538 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:10.518057 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:10.518502 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:10.518507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:10.518511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:10.518733 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:10.519380 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:30:10.519389 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:30:10.519810 (+ 421us) client_negotiation.cc:769] Sending connection context
0929 11:30:10.520088 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 11:30:10.520707 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":364,"threads_started":1}
I20260929 11:30:10.525135 6491 heartbeater.cc:347] Connected to a master server at 127.23.153.62:44055
I20260929 11:30:10.525578 6491 heartbeater.cc:464] Registering TS with master...
I20260929 11:30:10.526666 6491 heartbeater.cc:511] Master 127.23.153.62:44055 requested a full tablet report, sending...
I20260929 11:30:10.529006 6174 ts_manager.cc:194] Registered new tserver with Master: 552adac140c443bb9554d1537a14d779 (127.23.153.2:42549)
I20260929 11:30:10.529899 6174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:53557
W20260929 11:30:10.868957 6494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:10.869270 6494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:10.869390 6494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:10.877324 6494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:10.877462 6494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:10.877614 6494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:30:10.888438 6494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:40885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=37449
--tserver_master_addrs=127.23.153.62:44055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:10.889907 6494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:10.891387 6494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:10.907730 6502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:10.907760 6499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:10.907788 6500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:10.908624 6494 server_base.cc:1034] running on GCE node
I20260929 11:30:10.909302 6494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:10.910394 6494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:10.911635 6494 hybrid_clock.cc:648] HybridClock initialized: now 1790681410911571 us; error 64 us; skew 500 ppm
I20260929 11:30:10.911984 6494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:10 dist-test-slave-qk05 krb5kdc[5587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681410, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:10.915977 6494 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:30:10.917670 6494 webserver.cc:460] Webserver started at http://127.23.153.3:37449/ using document root <none> and password file <none>
I20260929 11:30:10.918427 6494 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:10.918560 6494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:10.926594 6494 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s
I20260929 11:30:10.930886 6509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:10.932879 6494 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:30:10.933110 6494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "4c7d1b658b4242f18129bded75a929df"
format_stamp: "Formatted at 2026-09-29 11:30:07 on dist-test-slave-qk05"
server_key: "6e785a7ab3d929a4c6294ecfddc83a76"
server_key_iv: "9365431a44faefd3b7e73dbb2e95346e"
server_key_version: "encryptionkey@0"
I20260929 11:30:10.934165 6494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:10.970746 6494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:10.976835 6494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:10.977115 6494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:10.978247 6494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:30:10.980044 6494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:30:10.980207 6494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:10.980326 6494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:30:10.980499 6494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:11.015782 6494 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:40885
I20260929 11:30:11.015843 6621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:40885 every 8 connection(s)
I20260929 11:30:11.017381 6494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:30:11.017951 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 6494
I20260929 11:30:11.021775 6359 heartbeater.cc:503] Master 127.23.153.62:44055 was elected leader, sending a full tablet report...
Sep 29 11:30:11 dist-test-slave-qk05 krb5kdc[5587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681410, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:30:11.036644 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.019082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45934 (local address 127.23.153.62:44055)
0929 11:30:11.019293 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:11.019300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:11.019824 (+ 524us) server_negotiation.cc:407] Connection header received
0929 11:30:11.020021 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:11.020028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:11.020121 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:11.020225 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:11.023207 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.024383 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.026533 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.027183 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.027321 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:11.030289 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:11.030296 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:11.030302 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:11.030350 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:11.033428 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:11.034459 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:11.034466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:11.034470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:11.034653 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:11.035111 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:11.035117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:11.035120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:11.035529 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:11.035814 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:11.036139 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 11:30:11.036343 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 11:30:11.043619 6625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.020995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:47773 (local address 127.23.153.62:44055)
0929 11:30:11.021802 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:11.021807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:11.023263 (+ 1456us) server_negotiation.cc:407] Connection header received
0929 11:30:11.026092 (+ 2829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:11.026097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:11.026198 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:11.026290 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:11.028492 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.029269 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.031308 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.031810 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.031938 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:11.036206 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:11.036212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:11.036217 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:11.036258 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:11.039967 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:11.040929 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:11.040933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:11.040947 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:11.041104 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:11.041575 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:11.041579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:11.041582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:11.041919 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:11.042055 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:11.042996 (+ 941us) server_negotiation.cc:299] Negotiation successful
0929 11:30:11.043288 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":474,"threads_started":1}
I20260929 11:30:11.045564 6626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.021673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:47773)
0929 11:30:11.022816 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:11.022885 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:11.025741 (+ 2856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:11.026625 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:11.026639 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:11.027493 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:30:11.028291 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:11.028316 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.029429 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.029433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:11.031158 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:11.031175 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.032060 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.032067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:11.032314 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:11.032915 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:30:11.032941 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:30:11.035913 (+ 2972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:30:11.040118 (+ 4205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:11.040127 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:11.040143 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:11.040768 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:11.041195 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:11.041199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:11.041201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:11.041429 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:11.042038 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:30:11.042045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:30:11.042430 (+ 385us) client_negotiation.cc:769] Sending connection context
0929 11:30:11.042949 (+ 519us) client_negotiation.cc:240] Negotiation successful
0929 11:30:11.043531 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":372,"threads_started":1}
I20260929 11:30:11.047891 6622 heartbeater.cc:347] Connected to a master server at 127.23.153.62:44055
I20260929 11:30:11.048264 6622 heartbeater.cc:464] Registering TS with master...
I20260929 11:30:11.049116 6622 heartbeater.cc:511] Master 127.23.153.62:44055 requested a full tablet report, sending...
I20260929 11:30:11.051095 6174 ts_manager.cc:194] Registered new tserver with Master: 4c7d1b658b4242f18129bded75a929df (127.23.153.3:40885)
I20260929 11:30:11.052119 6174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:47773
I20260929 11:30:11.052716 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:30:11.061134 6625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.053785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45946 (local address 127.23.153.62:44055)
0929 11:30:11.053941 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:11.053946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:11.055630 (+ 1684us) server_negotiation.cc:407] Connection header received
0929 11:30:11.055849 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:11.055853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:11.056003 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:11.056145 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:30:11.056578 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.057284 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.058794 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.059300 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.059422 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:11.059954 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:30:11.060504 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:30:11.060637 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:11.060755 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 11:30:11.060847 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 11:30:11.062181 6174 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:30:11.069638 6174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:30:11.110119 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.082875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53396 (local address 127.23.153.3:40885)
0929 11:30:11.100226 (+ 17351us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:11.100235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:11.100276 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:30:11.100415 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:11.100424 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:11.100733 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:11.102318 (+ 1585us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:11.102634 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.104032 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.107100 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.108720 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.108899 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:11.109174 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:11.109511 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 11:30:11.109704 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17136,"thread_start_us":17958,"threads_started":1}
I20260929 11:30:11.110177 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.082622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:40885 (local address 127.0.0.1:53396)
0929 11:30:11.097906 (+ 15284us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:11.097967 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:11.098211 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:11.101093 (+ 2882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:11.101104 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:11.101368 (+ 264us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:11.101962 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:11.101982 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.104214 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.104219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:11.106820 (+ 2601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:11.106839 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.109048 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.109055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:11.109276 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:11.109316 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 11:30:11.109439 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 11:30:11.109657 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14914,"mutex_wait_us":1,"thread_start_us":15806,"threads_started":1}
I20260929 11:30:11.111343 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.082869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:42549 (local address 127.0.0.1:60738)
0929 11:30:11.102187 (+ 19318us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:11.102215 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:11.102375 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:11.103639 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:11.103643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:11.103666 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:11.104122 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:11.104131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.105486 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.105490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:11.107802 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:11.107816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.110473 (+ 2657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.110480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:11.110680 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:11.110714 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 11:30:11.110811 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 11:30:11.110967 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19187,"thread_start_us":19433,"threads_started":1}
I20260929 11:30:11.111392 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.083500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.23.153.2:42549)
0929 11:30:11.099623 (+ 16123us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:11.099632 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:11.102541 (+ 2909us) server_negotiation.cc:407] Connection header received
0929 11:30:11.102666 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:11.102676 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:11.102978 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:11.103226 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:11.104305 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.105327 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.107969 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.109425 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.109582 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:11.110760 (+ 1178us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:11.110848 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 11:30:11.111053 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15951,"thread_start_us":15829,"threads_started":1}
I20260929 11:30:11.112856 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.084372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.23.153.1:44515)
0929 11:30:11.101659 (+ 17287us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:11.101670 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:11.102805 (+ 1135us) server_negotiation.cc:407] Connection header received
0929 11:30:11.102916 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:11.102922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:11.103151 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:11.103321 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:11.104296 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.105860 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.108590 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.110081 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.110220 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:11.110766 (+ 546us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:11.112343 (+ 1577us) server_negotiation.cc:299] Negotiation successful
0929 11:30:11.112534 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17115,"thread_start_us":16956,"threads_started":1}
I20260929 11:30:11.113423 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.083590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:44515 (local address 127.0.0.1:44030)
0929 11:30:11.102563 (+ 18973us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:11.102580 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:11.102690 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:11.103560 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:11.103566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:11.103593 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:11.104075 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:11.104089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.106066 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.106073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:11.108377 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:11.108394 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.110920 (+ 2526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.110926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:11.111138 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:11.111173 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:30:11.111261 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 11:30:11.113077 (+ 1816us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18843,"spinlock_wait_cycles":1920,"thread_start_us":18286,"threads_started":1}
I20260929 11:30:11.115254 6557 tablet_service.cc:1467] Processing CreateTablet for tablet 31029c02c4eb41259e503e2697a1e8ee (DEFAULT_TABLE table=test-table [id=1f1b5f85d79c4720ac23b8256d954648]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:30:11.117003 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31029c02c4eb41259e503e2697a1e8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:30:11.116503 6426 tablet_service.cc:1467] Processing CreateTablet for tablet 31029c02c4eb41259e503e2697a1e8ee (DEFAULT_TABLE table=test-table [id=1f1b5f85d79c4720ac23b8256d954648]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:30:11.116729 6294 tablet_service.cc:1467] Processing CreateTablet for tablet 31029c02c4eb41259e503e2697a1e8ee (DEFAULT_TABLE table=test-table [id=1f1b5f85d79c4720ac23b8256d954648]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:30:11.117975 6294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31029c02c4eb41259e503e2697a1e8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:30:11.117966 6426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31029c02c4eb41259e503e2697a1e8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:30:11.134295 6638 tablet_bootstrap.cc:492] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851: Bootstrap starting.
I20260929 11:30:11.136812 6639 tablet_bootstrap.cc:492] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df: Bootstrap starting.
I20260929 11:30:11.138974 6640 tablet_bootstrap.cc:492] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779: Bootstrap starting.
I20260929 11:30:11.140259 6638 tablet_bootstrap.cc:654] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851: Neither blocks nor log segments found. Creating new log.
I20260929 11:30:11.141095 6639 tablet_bootstrap.cc:654] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df: Neither blocks nor log segments found. Creating new log.
I20260929 11:30:11.142145 6638 log.cc:824] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:11.142982 6640 tablet_bootstrap.cc:654] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779: Neither blocks nor log segments found. Creating new log.
I20260929 11:30:11.142936 6639 log.cc:824] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:11.144428 6640 log.cc:824] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:11.148043 6640 tablet_bootstrap.cc:492] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779: No bootstrap required, opened a new log
I20260929 11:30:11.148085 6638 tablet_bootstrap.cc:492] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851: No bootstrap required, opened a new log
I20260929 11:30:11.148137 6639 tablet_bootstrap.cc:492] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df: No bootstrap required, opened a new log
I20260929 11:30:11.148382 6640 ts_tablet_manager.cc:1397] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260929 11:30:11.148391 6639 ts_tablet_manager.cc:1397] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 11:30:11.148391 6638 ts_tablet_manager.cc:1397] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.001s
I20260929 11:30:11.156774 6640 raft_consensus.cc:348] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } } peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } }
I20260929 11:30:11.157240 6640 raft_consensus.cc:374] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:30:11.157339 6640 raft_consensus.cc:729] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 552adac140c443bb9554d1537a14d779, State: Initialized, Role: FOLLOWER
I20260929 11:30:11.158062 6640 consensus_queue.cc:260] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } } peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } }
I20260929 11:30:11.159287 6491 heartbeater.cc:503] Master 127.23.153.62:44055 was elected leader, sending a full tablet report...
I20260929 11:30:11.160130 6640 ts_tablet_manager.cc:1428] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260929 11:30:11.159896 6639 raft_consensus.cc:348] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } } peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } }
I20260929 11:30:11.159896 6638 raft_consensus.cc:348] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } } peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } }
I20260929 11:30:11.160398 6639 raft_consensus.cc:374] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:30:11.160398 6638 raft_consensus.cc:374] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:30:11.160498 6639 raft_consensus.cc:729] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c7d1b658b4242f18129bded75a929df, State: Initialized, Role: FOLLOWER
I20260929 11:30:11.160498 6638 raft_consensus.cc:729] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a82470a638a141f0ade33721fd933851, State: Initialized, Role: FOLLOWER
I20260929 11:30:11.161222 6638 consensus_queue.cc:260] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } } peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } }
I20260929 11:30:11.161222 6639 consensus_queue.cc:260] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } } peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } }
I20260929 11:30:11.163959 6639 ts_tablet_manager.cc:1428] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260929 11:30:11.164069 6638 ts_tablet_manager.cc:1428] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260929 11:30:11.165203 6622 heartbeater.cc:503] Master 127.23.153.62:44055 was elected leader, sending a full tablet report...
W20260929 11:30:11.232815 6360 tablet.cc:2369] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:30:11.236030 6645 raft_consensus.cc:482] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:30:11.236368 6645 raft_consensus.cc:504] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } } peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } }
I20260929 11:30:11.240197 6645 leader_election.cc:288] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c7d1b658b4242f18129bded75a929df (127.23.153.3:40885), 552adac140c443bb9554d1537a14d779 (127.23.153.2:42549)
W20260929 11:30:11.246387 6492 tablet.cc:2369] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:30:11.248245 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.239455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:34041 (local address 127.23.153.2:42549)
0929 11:30:11.239840 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:11.239846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:11.240202 (+ 356us) server_negotiation.cc:407] Connection header received
0929 11:30:11.240474 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:11.240479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:11.240558 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:11.240621 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:11.241338 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.242466 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.245740 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.247109 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.247284 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:11.247361 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:11.247828 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 11:30:11.247969 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260929 11:30:11.248239 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.239617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:41795 (local address 127.23.153.3:40885)
0929 11:30:11.239824 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:11.239830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:11.240724 (+ 894us) server_negotiation.cc:407] Connection header received
0929 11:30:11.240963 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:11.240970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:11.241037 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:11.241116 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:11.241647 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.242724 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.245737 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.247190 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.247333 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:11.247403 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:11.247829 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 11:30:11.247969 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 11:30:11.248508 6648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.239353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:40885 (local address 127.23.153.1:41795)
0929 11:30:11.240636 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:11.240653 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:11.240816 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:11.241169 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:11.241174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:11.241196 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:11.241499 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:11.241507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.242977 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.242987 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:11.245513 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:11.245536 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.247406 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.247412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:11.247632 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:11.247683 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 11:30:11.247786 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 11:30:11.247972 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1161,"mutex_wait_us":72,"thread_start_us":994,"threads_started":1}
I20260929 11:30:11.248731 6647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.239231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:42549 (local address 127.23.153.1:34041)
0929 11:30:11.240096 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:11.240115 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:11.240305 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:11.240826 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:11.240831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:11.240852 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:11.241188 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:11.241199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.242626 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.242630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:11.245513 (+ 2883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:11.245536 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.247371 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.247381 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:11.247639 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:11.247691 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 11:30:11.247798 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 11:30:11.248011 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":134,"thread_start_us":495,"threads_started":1}
I20260929 11:30:11.249604 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31029c02c4eb41259e503e2697a1e8ee" candidate_uuid: "a82470a638a141f0ade33721fd933851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "552adac140c443bb9554d1537a14d779" is_pre_election: true
I20260929 11:30:11.249614 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31029c02c4eb41259e503e2697a1e8ee" candidate_uuid: "a82470a638a141f0ade33721fd933851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c7d1b658b4242f18129bded75a929df" is_pre_election: true
I20260929 11:30:11.250077 6577 raft_consensus.cc:2454] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a82470a638a141f0ade33721fd933851 in term 0.
I20260929 11:30:11.250074 6446 raft_consensus.cc:2454] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a82470a638a141f0ade33721fd933851 in term 0.
I20260929 11:30:11.250775 6248 leader_election.cc:302] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 552adac140c443bb9554d1537a14d779, a82470a638a141f0ade33721fd933851; no voters:
I20260929 11:30:11.251310 6645 raft_consensus.cc:2781] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:30:11.251492 6645 raft_consensus.cc:482] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:30:11.251605 6645 raft_consensus.cc:3037] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:30:11.253777 6645 raft_consensus.cc:504] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } } peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } }
I20260929 11:30:11.254382 6645 leader_election.cc:288] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [CANDIDATE]: Term 1 election: Requested vote from peers 4c7d1b658b4242f18129bded75a929df (127.23.153.3:40885), 552adac140c443bb9554d1537a14d779 (127.23.153.2:42549)
I20260929 11:30:11.254941 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31029c02c4eb41259e503e2697a1e8ee" candidate_uuid: "a82470a638a141f0ade33721fd933851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c7d1b658b4242f18129bded75a929df"
I20260929 11:30:11.254969 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31029c02c4eb41259e503e2697a1e8ee" candidate_uuid: "a82470a638a141f0ade33721fd933851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "552adac140c443bb9554d1537a14d779"
I20260929 11:30:11.255196 6446 raft_consensus.cc:3037] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:30:11.255196 6577 raft_consensus.cc:3037] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:30:11.257421 6446 raft_consensus.cc:2454] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a82470a638a141f0ade33721fd933851 in term 1.
I20260929 11:30:11.257439 6577 raft_consensus.cc:2454] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a82470a638a141f0ade33721fd933851 in term 1.
I20260929 11:30:11.258122 6249 leader_election.cc:302] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4c7d1b658b4242f18129bded75a929df, 552adac140c443bb9554d1537a14d779, a82470a638a141f0ade33721fd933851; no voters:
I20260929 11:30:11.258486 6645 raft_consensus.cc:2781] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:30:11.258945 6645 raft_consensus.cc:686] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [term 1 LEADER]: Becoming Leader. State: Replica: a82470a638a141f0ade33721fd933851, State: Running, Role: LEADER
I20260929 11:30:11.259521 6645 consensus_queue.cc:237] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } } peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } }
I20260929 11:30:11.264197 6172 catalog_manager.cc:5495] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 reported cstate change: term changed from 0 to 1, leader changed from <none> to a82470a638a141f0ade33721fd933851 (127.23.153.1). New cstate: current_term: 1 leader_uuid: "a82470a638a141f0ade33721fd933851" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a82470a638a141f0ade33721fd933851" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 44515 } health_report { overall_health: HEALTHY } } }
W20260929 11:30:11.269200 6623 tablet.cc:2369] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:30:11.283568 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.277202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44040 (local address 127.23.153.1:44515)
0929 11:30:11.277452 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:11.277458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:11.277476 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:30:11.277579 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:11.277583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:11.277895 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:11.278091 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:30:11.278504 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.279311 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.280771 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.281205 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.281327 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:11.281914 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:30:11.282450 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:30:11.282832 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:11.283060 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 11:30:11.283198 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 11:30:11.293953 6446 raft_consensus.cc:1261] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779 [term 1 FOLLOWER]: Refusing update from remote peer a82470a638a141f0ade33721fd933851: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:30:11.294068 6577 raft_consensus.cc:1261] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df [term 1 FOLLOWER]: Refusing update from remote peer a82470a638a141f0ade33721fd933851: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:30:11.295238 6645 consensus_queue.cc:1035] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "552adac140c443bb9554d1537a14d779" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 42549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:30:11.295779 6649 consensus_queue.cc:1035] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c7d1b658b4242f18129bded75a929df" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:30:11.323642 6173 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:30:11.323961 6173 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:30:11.328474 6173 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402: Sending DeleteTablet for 3 replicas of tablet 31029c02c4eb41259e503e2697a1e8ee
I20260929 11:30:11.329941 6294 tablet_service.cc:1514] Processing DeleteTablet for tablet 31029c02c4eb41259e503e2697a1e8ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:30:11 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:44030
I20260929 11:30:11.329946 6426 tablet_service.cc:1514] Processing DeleteTablet for tablet 31029c02c4eb41259e503e2697a1e8ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:30:11 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:60738
I20260929 11:30:11.329927 6557 tablet_service.cc:1514] Processing DeleteTablet for tablet 31029c02c4eb41259e503e2697a1e8ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:30:11 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:53396
I20260929 11:30:11.333117 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 6211
I20260929 11:30:11.343899 6663 tablet_replica.cc:330] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779: stopping tablet replica
I20260929 11:30:11.344676 6663 raft_consensus.cc:2229] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:30:11.345441 6663 raft_consensus.cc:2258] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:30:11.346487 6662 tablet_replica.cc:330] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851: stopping tablet replica
I20260929 11:30:11.346875 6664 tablet_replica.cc:330] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df: stopping tablet replica
I20260929 11:30:11.347092 6662 raft_consensus.cc:2229] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:30:11.347693 6664 raft_consensus.cc:2229] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:30:11.348109 6662 raft_consensus.cc:2258] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:30:11.348733 6664 raft_consensus.cc:2258] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:30:11.348953 6663 ts_tablet_manager.cc:1905] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:30:11.350772 6662 ts_tablet_manager.cc:1905] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:30:11.355553 6664 ts_tablet_manager.cc:1905] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:30:11.356732 6663 ts_tablet_manager.cc:1918] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:30:11.356932 6663 log.cc:1196] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/31029c02c4eb41259e503e2697a1e8ee
I20260929 11:30:11.357532 6663 ts_tablet_manager.cc:1939] T 31029c02c4eb41259e503e2697a1e8ee P 552adac140c443bb9554d1537a14d779: Deleting consensus metadata
I20260929 11:30:11.359762 6159 catalog_manager.cc:4841] TS 552adac140c443bb9554d1537a14d779 (127.23.153.2:42549): tablet 31029c02c4eb41259e503e2697a1e8ee (table test-table [id=1f1b5f85d79c4720ac23b8256d954648]) successfully deleted
I20260929 11:30:11.361786 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.334172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44048 (local address 127.23.153.1:44515)
0929 11:30:11.334358 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:11.334364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:11.334475 (+ 111us) server_negotiation.cc:407] Connection header received
0929 11:30:11.334697 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:11.334704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:11.334799 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:11.334901 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:11.336752 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.338102 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.340373 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.341170 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.341326 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:11.352268 (+ 10942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:11.352286 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:11.352306 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:11.352362 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:11.357348 (+ 4986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:11.358546 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:11.358557 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:11.358561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:11.358886 (+ 325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:11.359462 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:11.359468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:11.359472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:11.360014 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:11.360357 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:11.361073 (+ 716us) server_negotiation.cc:299] Negotiation successful
0929 11:30:11.361368 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:30:11.362711 6354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:11.364349 6664 ts_tablet_manager.cc:1918] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:30:11.364571 6664 log.cc:1196] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/31029c02c4eb41259e503e2697a1e8ee
I20260929 11:30:11.365283 6664 ts_tablet_manager.cc:1939] T 31029c02c4eb41259e503e2697a1e8ee P 4c7d1b658b4242f18129bded75a929df: Deleting consensus metadata
I20260929 11:30:11.368541 6160 catalog_manager.cc:4841] TS 4c7d1b658b4242f18129bded75a929df (127.23.153.3:40885): tablet 31029c02c4eb41259e503e2697a1e8ee (table test-table [id=1f1b5f85d79c4720ac23b8256d954648]) successfully deleted
I20260929 11:30:11.468030 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 6211
I20260929 11:30:11.485885 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 6363
W20260929 11:30:11.485931 6161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.153.1:44515 (error 108)
I20260929 11:30:11.488212 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.487606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:44515 (local address 127.0.0.1:44064)
0929 11:30:11.487886 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:11.488024 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260929 11:30:11.488483 6161 catalog_manager.cc:4584] TS a82470a638a141f0ade33721fd933851 (127.23.153.1:44515): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31029c02c4eb41259e503e2697a1e8ee: Network error: Client connection negotiation failed: client connection to 127.23.153.1:44515: connect: Connection refused (error 111)
I20260929 11:30:11.504114 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.486971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.23.153.2:42549)
0929 11:30:11.487215 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:11.487220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:11.487235 (+ 15us) server_negotiation.cc:407] Connection header received
0929 11:30:11.487289 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:11.487293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:11.487355 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:11.487476 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:11.489505 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.490556 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.492288 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.492775 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.492897 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:11.496464 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:11.496478 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:11.496493 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:11.496539 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:11.499757 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:11.500919 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:11.500929 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:11.500933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:11.501105 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:11.501554 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:11.501561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:11.501565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:11.502492 (+ 927us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:11.502962 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:11.503273 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 11:30:11.503665 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260929 11:30:11.504963 6486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:11.542210 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.541613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:44515 (local address 127.0.0.1:44066)
0929 11:30:11.541857 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:11.541992 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 11:30:11.543390 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.542945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:44515 (local address 127.0.0.1:44074)
0929 11:30:11.543137 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:11.543246 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 11:30:11.598850 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 6363
I20260929 11:30:11.604152 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.603582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:44515 (local address 127.0.0.1:44090)
0929 11:30:11.603817 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:11.603929 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 11:30:11.605324 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.604917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:44515 (local address 127.0.0.1:44098)
0929 11:30:11.605090 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:11.605180 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 11:30:11.614596 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 6494
I20260929 11:30:11.629281 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.615423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53402 (local address 127.23.153.3:40885)
0929 11:30:11.615604 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:11.615608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:11.615629 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:30:11.615825 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:11.615829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:11.615908 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:11.616007 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:11.617424 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.618183 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.619687 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:11.620104 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:11.620232 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:11.622996 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:11.623059 (+ 63us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:11.623083 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:11.623134 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:11.626253 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:11.627245 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:11.627254 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:11.627256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:11.627429 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:11.627813 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:11.627816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:11.627818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:11.628220 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:11.628458 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:11.628716 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 11:30:11.628914 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 11:30:11.629972 6617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:11.686139 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.685625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:44515 (local address 127.0.0.1:44112)
0929 11:30:11.685871 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:11.685971 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 11:30:11.687299 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:11.686867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:44515 (local address 127.0.0.1:44116)
0929 11:30:11.687099 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:11.687178 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 11:30:11.724031 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 6494
I20260929 11:30:11.738036 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 6141
I20260929 11:30:11.739789 6204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:11.856454 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 6141
I20260929 11:30:11.876639 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:44055
--webserver_interface=127.23.153.62
--webserver_port=39609
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:44055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:30:12.243103 6670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:12.243459 6670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:12.243600 6670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:12.251808 6670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:30:12.251967 6670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:12.252045 6670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:12.252126 6670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:30:12.252192 6670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:30:12.264127 6670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:44055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:44055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=39609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:12.265674 6670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:12.267155 6670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:12.280110 6675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:12.280347 6678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:12.280162 6676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:12.281344 6670 server_base.cc:1034] running on GCE node
I20260929 11:30:12.282205 6670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:12.283943 6670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:12.285197 6670 hybrid_clock.cc:648] HybridClock initialized: now 1790681412285144 us; error 44 us; skew 500 ppm
I20260929 11:30:12.285620 6670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:12 dist-test-slave-qk05 krb5kdc[5587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681412, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:12.289278 6670 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:30:12.291054 6670 webserver.cc:460] Webserver started at http://127.23.153.62:39609/ using document root <none> and password file <none>
I20260929 11:30:12.292057 6670 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:12.292223 6670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:12.299574 6670 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 11:30:12.303244 6685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:12.305015 6670 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:30:12.305272 6670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "076d6020653b415c9439019df315f402"
format_stamp: "Formatted at 2026-09-29 11:30:06 on dist-test-slave-qk05"
server_key: "03ac5bb2b8d499ca591bec6422281109"
server_key_iv: "b76f1e61989ea99354821039a1af583e"
server_key_version: "encryptionkey@0"
I20260929 11:30:12.306370 6670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:12.328581 6670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:12.335429 6670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:12.335778 6670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:12.359119 6670 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:44055
I20260929 11:30:12.359119 6736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:44055 every 8 connection(s)
I20260929 11:30:12.361047 6670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:30:12.363833 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 6670
I20260929 11:30:12.364880 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:44515
--local_ip_for_outbound_sockets=127.23.153.1
--tserver_master_addrs=127.23.153.62:44055
--webserver_port=40177
--webserver_interface=127.23.153.1
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:30:12.371261 6737 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:30:12.375861 6737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402: Bootstrap starting.
I20260929 11:30:12.399260 6737 log.cc:824] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:12.418401 6737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:30:12.419049 6737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402: Bootstrap complete.
I20260929 11:30:12.428886 6737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076d6020653b415c9439019df315f402" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44055 } }
I20260929 11:30:12.429859 6737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 076d6020653b415c9439019df315f402, State: Initialized, Role: FOLLOWER
I20260929 11:30:12.430531 6737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076d6020653b415c9439019df315f402" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44055 } }
I20260929 11:30:12.430964 6737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:30:12.431149 6737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:30:12.431335 6737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 2 FOLLOWER]: Advancing to term 3
I20260929 11:30:12.435609 6737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076d6020653b415c9439019df315f402" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44055 } }
I20260929 11:30:12.436224 6737 leader_election.cc:302] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 076d6020653b415c9439019df315f402; no voters:
I20260929 11:30:12.436897 6737 leader_election.cc:288] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 11:30:12.437043 6741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 11:30:12.438532 6741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [term 3 LEADER]: Becoming Leader. State: Replica: 076d6020653b415c9439019df315f402, State: Running, Role: LEADER
I20260929 11:30:12.439379 6741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076d6020653b415c9439019df315f402" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44055 } }
I20260929 11:30:12.439971 6737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:30:12.443313 6743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 076d6020653b415c9439019df315f402. Latest consensus state: current_term: 3 leader_uuid: "076d6020653b415c9439019df315f402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076d6020653b415c9439019df315f402" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44055 } } }
I20260929 11:30:12.443517 6742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "076d6020653b415c9439019df315f402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "076d6020653b415c9439019df315f402" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 44055 } } }
I20260929 11:30:12.443819 6743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [sys.catalog]: This master's current role is: LEADER
I20260929 11:30:12.443900 6742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 076d6020653b415c9439019df315f402 [sys.catalog]: This master's current role is: LEADER
I20260929 11:30:12.456019 6750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:30:12.466132 6750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:30:12.468026 6750 catalog_manager.cc:1259] Loaded cluster ID: 53863fcc9f3c46b2bb4eb3860d0f5679
I20260929 11:30:12.468148 6750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 11:30:12.470783 6750 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa3b4e789ad google::LogMessage::Fail() at ??:0
@ 0x7fa3b4e7b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa3b4e784de google::LogMessage::Flush() at ??:0
@ 0x7fa3b4e79a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa3ca8ab512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa3ca8a8976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa3ca930610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa3ca930439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa3b7ac4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa3b7ac791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa3b7ac7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa3b7aa2eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa3bad7c6db start_thread at ??:0
@ 0x7fa3b3d1261f clone at ??:0
W20260929 11:30:12.800226 6739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:12.800565 6739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:12.800695 6739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:12.809283 6739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:12.809667 6739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:12.809854 6739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:30:12.822091 6739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:44515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=40177
--tserver_master_addrs=127.23.153.62:44055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:12.823663 6739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:12.825122 6739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:12.840205 6778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:12.840175 6781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:12.840842 6779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:12.841555 6739 server_base.cc:1034] running on GCE node
I20260929 11:30:12.842290 6739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:12.844089 6739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:12.845414 6739 hybrid_clock.cc:648] HybridClock initialized: now 1790681412845334 us; error 75 us; skew 500 ppm
I20260929 11:30:12.845989 6739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:12 dist-test-slave-qk05 krb5kdc[5587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681412, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:12.850593 6739 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:30:12.852553 6739 webserver.cc:460] Webserver started at http://127.23.153.1:40177/ using document root <none> and password file <none>
I20260929 11:30:12.853493 6739 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:12.853660 6739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:12.860018 6739 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:30:12.863752 6788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:12.865970 6739 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260929 11:30:12.866314 6739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "a82470a638a141f0ade33721fd933851"
format_stamp: "Formatted at 2026-09-29 11:30:06 on dist-test-slave-qk05"
server_key: "7455426161b4ecfe647a966478f54431"
server_key_iv: "59d5d28c7296d97b49608113bd58671f"
server_key_version: "encryptionkey@0"
I20260929 11:30:12.868120 6739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:12.887571 6739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:12.892545 6739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:12.892843 6739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:12.894119 6739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:30:12.896359 6795 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260929 11:30:12.902341 6795 ts_tablet_manager.cc:1852] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 11:30:12.903542 6795 ts_tablet_manager.cc:1905] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:30:12.914031 6795 ts_tablet_manager.cc:1918] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:30:12.914569 6795 log.cc:1196] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/31029c02c4eb41259e503e2697a1e8ee
I20260929 11:30:12.915472 6795 ts_tablet_manager.cc:1939] T 31029c02c4eb41259e503e2697a1e8ee P a82470a638a141f0ade33721fd933851: Deleting consensus metadata
I20260929 11:30:12.916280 6739 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 11:30:12.916430 6739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.020s user 0.001s sys 0.000s
I20260929 11:30:12.916577 6739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:30:12.916651 6739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:12.959103 6739 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:44515
I20260929 11:30:12.959141 6901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:44515 every 8 connection(s)
I20260929 11:30:12.960731 6739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:30:12.966622 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 6739
I20260929 11:30:12.967221 6904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:12.964073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:41785)
0929 11:30:12.965231 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:12.965787 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":341,"threads_started":1}
I20260929 11:30:12.967443 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:42549
--local_ip_for_outbound_sockets=127.23.153.2
--tserver_master_addrs=127.23.153.62:44055
--webserver_port=38607
--webserver_interface=127.23.153.2
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:30:12.967979 6902 heartbeater.cc:644] Failed to heartbeat to 127.23.153.62:44055 (0 consecutive failures): Network error: Failed to ping master at 127.23.153.62:44055: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
I20260929 11:30:12.969035 6904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:12.968582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:59995)
0929 11:30:12.968785 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:12.968884 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 11:30:12.970100 6904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:12.969686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:54613)
0929 11:30:12.969845 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:12.969941 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 11:30:12.970363 6902 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 11:30:13.330992 6905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:13.331391 6905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:13.331527 6905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:13.339241 6905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:13.339378 6905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:13.339541 6905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:30:13.350123 6905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:42549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=38607
--tserver_master_addrs=127.23.153.62:44055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:13.351769 6905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:13.353230 6905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:13.369115 6910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:13.369258 6911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:13.369426 6913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:13.369570 6905 server_base.cc:1034] running on GCE node
I20260929 11:30:13.370501 6905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:13.371667 6905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:13.372947 6905 hybrid_clock.cc:648] HybridClock initialized: now 1790681413372860 us; error 63 us; skew 500 ppm
I20260929 11:30:13.373322 6905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:13 dist-test-slave-qk05 krb5kdc[5587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681413, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:13.376955 6905 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:30:13.378587 6905 webserver.cc:460] Webserver started at http://127.23.153.2:38607/ using document root <none> and password file <none>
I20260929 11:30:13.379400 6905 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:13.379544 6905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:13.385815 6905 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:30:13.389217 6920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:13.391280 6905 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:30:13.391500 6905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "552adac140c443bb9554d1537a14d779"
format_stamp: "Formatted at 2026-09-29 11:30:07 on dist-test-slave-qk05"
server_key: "0b39ec96c4eb8e2c3d185b0350fc6f60"
server_key_iv: "89559dccfa9e7ec6a5aa3c11c129f7cd"
server_key_version: "encryptionkey@0"
I20260929 11:30:13.392616 6905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:13.409463 6905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:13.414844 6905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:13.415215 6905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:13.416337 6905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:30:13.417850 6905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:30:13.417964 6905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:13.418084 6905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:30:13.418157 6905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:13.452437 6905 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:42549
I20260929 11:30:13.452484 7032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:42549 every 8 connection(s)
I20260929 11:30:13.453991 6905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:30:13.455555 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 6905
I20260929 11:30:13.456539 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:40885
--local_ip_for_outbound_sockets=127.23.153.3
--tserver_master_addrs=127.23.153.62:44055
--webserver_port=37449
--webserver_interface=127.23.153.3
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:30:13.462031 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:13.458930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:46967)
0929 11:30:13.460285 (+ 1355us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:13.460602 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":958,"thread_start_us":646,"threads_started":1}
W20260929 11:30:13.463045 7033 heartbeater.cc:644] Failed to heartbeat to 127.23.153.62:44055 (0 consecutive failures): Network error: Failed to ping master at 127.23.153.62:44055: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
I20260929 11:30:13.464221 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:13.463765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:38795)
0929 11:30:13.463956 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:13.464051 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 11:30:13.465442 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:13.465022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:46745)
0929 11:30:13.465225 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:13.465313 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260929 11:30:13.465739 7033 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 11:30:13.832921 7036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:13.833254 7036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:13.833359 7036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:13.842764 7036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:13.842933 7036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:13.843118 7036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:30:13.854931 7036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:40885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=37449
--tserver_master_addrs=127.23.153.62:44055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:13.856403 7036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:13.857825 7036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:13.874411 7041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:13.874657 7042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:13.875233 7044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:13.876250 7036 server_base.cc:1034] running on GCE node
I20260929 11:30:13.877120 7036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:13.878232 7036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:13.879453 7036 hybrid_clock.cc:648] HybridClock initialized: now 1790681413879387 us; error 65 us; skew 500 ppm
I20260929 11:30:13.879930 7036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:13 dist-test-slave-qk05 krb5kdc[5587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681413, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:13.883657 7036 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:30:13.885329 7036 webserver.cc:460] Webserver started at http://127.23.153.3:37449/ using document root <none> and password file <none>
I20260929 11:30:13.886098 7036 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:13.886224 7036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:13.892316 7036 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 11:30:13.895614 7051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:13.897352 7036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 11:30:13.897572 7036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "4c7d1b658b4242f18129bded75a929df"
format_stamp: "Formatted at 2026-09-29 11:30:07 on dist-test-slave-qk05"
server_key: "6e785a7ab3d929a4c6294ecfddc83a76"
server_key_iv: "9365431a44faefd3b7e73dbb2e95346e"
server_key_version: "encryptionkey@0"
I20260929 11:30:13.899317 7036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:13.925793 7036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:13.931929 7036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:13.932238 7036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:13.933377 7036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:30:13.934957 7036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:30:13.935107 7036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 11:30:13.935235 7036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:30:13.935314 7036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:13.972386 7160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:13.971159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:58329)
0929 11:30:13.971791 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:13.972060 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":276,"threads_started":1}
I20260929 11:30:13.972910 7036 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:40885
I20260929 11:30:13.972965 7164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:40885 every 8 connection(s)
I20260929 11:30:13.974546 7036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:30:13.980314 7167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:13.977924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:48409)
0929 11:30:13.978808 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:13.979142 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":345,"threads_started":1}
W20260929 11:30:13.981513 7165 heartbeater.cc:644] Failed to heartbeat to 127.23.153.62:44055 (0 consecutive failures): Network error: Failed to ping master at 127.23.153.62:44055: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
I20260929 11:30:13.982647 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 7036
I20260929 11:30:13.982769 7167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:13.982337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:54221)
0929 11:30:13.982520 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:13.982621 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 11:30:13.984019 7167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:13.983447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:56379)
0929 11:30:13.983637 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:13.983848 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260929 11:30:13.984375 7165 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 11:30:14.467487 7169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:14.466499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:47725)
0929 11:30:14.467096 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:14.467245 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":277,"threads_started":1}
I20260929 11:30:14.974623 7170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:14.973666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:36965)
0929 11:30:14.974306 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:14.974414 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":337,"threads_started":1}
I20260929 11:30:14.986068 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:14.985128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:38373)
0929 11:30:14.985683 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:14.985811 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":262,"threads_started":1}
I20260929 11:30:15.469513 7172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:15.468622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:45729)
0929 11:30:15.469220 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:15.469313 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":353,"threads_started":1}
I20260929 11:30:15.977223 7173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:15.976034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:39615)
0929 11:30:15.976802 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:15.977008 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":371,"threads_started":1}
I20260929 11:30:15.988276 7174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:15.987432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:38559)
0929 11:30:15.988000 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:15.988093 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":289,"threads_started":1}
I20260929 11:30:16.471498 7175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:16.470653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:54455)
0929 11:30:16.471195 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:16.471294 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":271,"threads_started":1}
I20260929 11:30:16.979317 7176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:16.978473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:42529)
0929 11:30:16.978993 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:16.979134 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":252,"threads_started":1}
I20260929 11:30:16.990159 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:16.989367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:60839)
0929 11:30:16.989867 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:16.989961 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":262,"threads_started":1}
I20260929 11:30:17.474399 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:17.473419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:40829)
0929 11:30:17.474107 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:17.474195 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":434,"threads_started":1}
I20260929 11:30:17.981138 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:17.980365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:40455)
0929 11:30:17.980864 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:17.980959 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":254,"threads_started":1}
I20260929 11:30:17.992271 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:17.991327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:36081)
0929 11:30:17.991989 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:17.992089 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":263,"threads_started":1}
I20260929 11:30:18.476425 7181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:18.475545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:44751)
0929 11:30:18.476147 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:18.476247 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":336,"threads_started":1}
I20260929 11:30:18.983108 7182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:18.982156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:55225)
0929 11:30:18.982654 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:18.982764 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":257,"threads_started":1}
I20260929 11:30:18.994318 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:18.993406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:58803)
0929 11:30:18.994030 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:18.994126 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":256,"threads_started":1}
I20260929 11:30:19.478274 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:19.477442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:43857)
0929 11:30:19.477991 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:19.478102 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":247,"threads_started":1}
I20260929 11:30:19.985090 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:19.984142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:39993)
0929 11:30:19.984709 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:19.984829 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":268,"threads_started":1}
I20260929 11:30:19.996299 7186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:19.995504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:33191)
0929 11:30:19.996002 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:19.996115 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":257,"threads_started":1}
I20260929 11:30:20.480770 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:20.479557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:33905)
0929 11:30:20.480232 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:20.480358 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":289,"threads_started":1}
I20260929 11:30:20.986850 7188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:20.986069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:46647)
0929 11:30:20.986569 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:20.986666 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":280,"threads_started":1}
I20260929 11:30:20.998204 7189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:20.997349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:55477)
0929 11:30:20.997855 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:20.997974 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":208,"threads_started":1}
I20260929 11:30:21.482944 7190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:21.482092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:33363)
0929 11:30:21.482631 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:21.482747 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":261,"threads_started":1}
I20260929 11:30:21.988903 7191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:21.988086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:49225)
0929 11:30:21.988626 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:21.988728 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":284,"threads_started":1}
I20260929 11:30:22.000169 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:21.999339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:47823)
0929 11:30:21.999859 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:21.999955 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":270,"threads_started":1}
I20260929 11:30:22.484829 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:22.484001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:39957)
0929 11:30:22.484546 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:22.484641 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":271,"threads_started":1}
I20260929 11:30:22.991318 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:22.990237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:56999)
0929 11:30:22.990719 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:22.990813 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":265,"threads_started":1}
I20260929 11:30:23.002231 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:23.001228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:45847)
0929 11:30:23.001925 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:23.002042 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":338,"threads_started":1}
I20260929 11:30:23.486753 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:23.485923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:39477)
0929 11:30:23.486473 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:23.486572 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":301,"threads_started":1}
I20260929 11:30:23.993837 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:23.992493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:40339)
0929 11:30:23.993250 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:23.993399 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":421,"threads_started":1}
I20260929 11:30:24.004060 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:24.003309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:41279)
0929 11:30:24.003799 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:24.003894 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":272,"threads_started":1}
I20260929 11:30:24.488667 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:24.487864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:40371)
0929 11:30:24.488373 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:24.488481 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":246,"threads_started":1}
I20260929 11:30:24.996342 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:24.995344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:51581)
0929 11:30:24.995818 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:24.996097 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":242,"threads_started":1}
I20260929 11:30:25.006307 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:25.005235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:54943)
0929 11:30:25.005860 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:25.006103 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":311,"threads_started":1}
I20260929 11:30:25.490435 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:25.489693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:49437)
0929 11:30:25.490181 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:25.490272 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":258,"threads_started":1}
I20260929 11:30:25.998643 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:25.997705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:51913)
0929 11:30:25.998315 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:25.998436 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":364,"threads_started":1}
I20260929 11:30:26.008236 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:26.007460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:47809)
0929 11:30:26.007977 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:26.008067 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":279,"threads_started":1}
I20260929 11:30:26.492484 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:26.491606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:54409)
0929 11:30:26.492175 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:26.492288 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":335,"threads_started":1}
I20260929 11:30:27.000751 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:26.999877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:46589)
0929 11:30:27.000459 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:27.000543 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":270,"threads_started":1}
I20260929 11:30:27.010197 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:27.009352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:48035)
0929 11:30:27.009892 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:27.010003 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":256,"threads_started":1}
I20260929 11:30:27.494411 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:27.493589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:53569)
0929 11:30:27.494130 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:27.494222 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":224,"threads_started":1}
I20260929 11:30:28.002844 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:28.001999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:48837)
0929 11:30:28.002492 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:28.002593 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":252,"threads_started":1}
I20260929 11:30:28.012547 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:28.011688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:58177)
0929 11:30:28.012218 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:28.012334 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":326,"threads_started":1}
I20260929 11:30:28.497085 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:28.496011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.2:56823)
0929 11:30:28.496662 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:28.496832 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":323,"threads_started":1}
I20260929 11:30:28.987385 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 6739
I20260929 11:30:29.005028 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:29.003935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.1:37725)
0929 11:30:29.004573 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:29.004757 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":279,"threads_started":1}
I20260929 11:30:29.012158 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:28.988523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45698 (local address 127.23.153.1:44515)
0929 11:30:28.989182 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:28.989201 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:28.989249 (+ 48us) server_negotiation.cc:407] Connection header received
0929 11:30:28.991257 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:28.991272 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:28.991918 (+ 646us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:28.992290 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:28.993895 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:28.995376 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:28.997144 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:28.998980 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:28.999349 (+ 369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:29.002851 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:29.002865 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:29.002888 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:29.002951 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:29.006839 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:29.008232 (+ 1393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:29.008242 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:29.008251 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:29.008518 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:29.009171 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:29.009175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:29.009177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:29.009838 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:29.010131 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:29.011142 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0929 11:30:29.011821 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":300,"threads_started":1}
I20260929 11:30:29.014214 6897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:29.015400 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:29.014019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:44055 (local address 127.23.153.3:54009)
0929 11:30:29.014864 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:29.015117 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:44055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":346,"threads_started":1}
I20260929 11:30:29.109660 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 6739
I20260929 11:30:29.126806 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 6905
I20260929 11:30:29.147928 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:29.128015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50690 (local address 127.23.153.2:42549)
0929 11:30:29.128579 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:29.128594 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:29.128631 (+ 37us) server_negotiation.cc:407] Connection header received
0929 11:30:29.130240 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:29.130261 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:29.130724 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:29.131162 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:29.133198 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:29.134895 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:29.136599 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:29.137452 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:29.137630 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:29.141025 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:29.141036 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:29.141055 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:29.141107 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:29.144231 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:29.145085 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:29.145093 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:29.145101 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:29.145308 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:29.145719 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:29.145723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:29.145726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:29.146382 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:29.146684 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:29.147192 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 11:30:29.147615 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":287,"threads_started":1}
I20260929 11:30:29.149442 7028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:29.238369 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 6905
I20260929 11:30:29.254452 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 7036
I20260929 11:30:29.276942 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:29.255983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58442 (local address 127.23.153.3:40885)
0929 11:30:29.256799 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:29.256815 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:29.256862 (+ 47us) server_negotiation.cc:407] Connection header received
0929 11:30:29.258593 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:29.258605 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:29.259190 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:29.259569 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:29.261232 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:29.263060 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:29.264688 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:29.265525 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:29.265700 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:29.268862 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:29.268873 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:29.268893 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:29.269055 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:29.272391 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:29.273293 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:29.273301 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:29.273309 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:29.273528 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:29.274452 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:29.274458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:29.274460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:29.275263 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:29.275558 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:29.276174 (+ 616us) server_negotiation.cc:299] Negotiation successful
0929 11:30:29.276592 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":382,"threads_started":1}
I20260929 11:30:29.278450 7159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:29.370028 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 7036
2026-09-29T11:30:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24955 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:30:29 dist-test-slave-qk05 krb5kdc[7222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:30:29 dist-test-slave-qk05 krb5kdc[7222](info): set up 2 sockets
Sep 29 11:30:29 dist-test-slave-qk05 krb5kdc[7222](info): commencing operation
krb5kdc: starting...
W20260929 11:30:30.431981 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:30:30 dist-test-slave-qk05 krb5kdc[7222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:30:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:30:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:30:30.695736 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:37051
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:37051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:30:31.064412 7235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:31.064749 7235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:31.064841 7235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:31.072454 7235 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 11:30:31.072585 7235 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 11:30:31.072623 7235 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 11:30:31.072667 7235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:30:31.072726 7235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:31.072795 7235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:31.072840 7235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:30:31.072901 7235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:30:31.083252 7235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:37051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:37051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:31.084614 7235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:31.085983 7235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:31.099292 7240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:31.099313 7241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:31.099351 7243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:31.099877 7235 server_base.cc:1034] running on GCE node
I20260929 11:30:31.100844 7235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:31.102761 7235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:31.104174 7235 hybrid_clock.cc:648] HybridClock initialized: now 1790681431104072 us; error 71 us; skew 500 ppm
I20260929 11:30:31.104645 7235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:31 dist-test-slave-qk05 krb5kdc[7222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681431, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:31.113164 7235 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:30:31.114949 7235 webserver.cc:460] Webserver started at http://127.23.153.62:41413/ using document root <none> and password file <none>
I20260929 11:30:31.115797 7235 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:31.115926 7235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:31.116266 7235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:30:31.119719 7235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "e31d2ca90b2c4b07a097a1b02f45e19a"
format_stamp: "Formatted at 2026-09-29 11:30:31 on dist-test-slave-qk05"
server_key: "f47b31f89441aa9350e8be5efeaf7db3"
server_key_iv: "a4447f83ee612b943d76eaf18913544e"
server_key_version: "encryptionkey@0"
I20260929 11:30:31.120862 7235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "e31d2ca90b2c4b07a097a1b02f45e19a"
format_stamp: "Formatted at 2026-09-29 11:30:31 on dist-test-slave-qk05"
server_key: "f47b31f89441aa9350e8be5efeaf7db3"
server_key_iv: "a4447f83ee612b943d76eaf18913544e"
server_key_version: "encryptionkey@0"
I20260929 11:30:31.127272 7235 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 11:30:31.131515 7250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:31.133169 7235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:30:31.133409 7235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "e31d2ca90b2c4b07a097a1b02f45e19a"
format_stamp: "Formatted at 2026-09-29 11:30:31 on dist-test-slave-qk05"
server_key: "f47b31f89441aa9350e8be5efeaf7db3"
server_key_iv: "a4447f83ee612b943d76eaf18913544e"
server_key_version: "encryptionkey@0"
I20260929 11:30:31.133664 7235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:31.150954 7235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:31.155833 7235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:31.156124 7235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:31.176777 7235 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:37051
I20260929 11:30:31.176800 7301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:37051 every 8 connection(s)
I20260929 11:30:31.178252 7235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:30:31.181672 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 7235
I20260929 11:30:31.181879 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:30:31.182504 24164 external_mini_cluster.cc:1428] Setting key de511bd2be6b80b97ac29474d4855799
I20260929 11:30:31.182631 7302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:30:31.197960 7302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: Bootstrap starting.
Sep 29 11:30:31 dist-test-slave-qk05 krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:30:31.202008 7302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: Neither blocks nor log segments found. Creating new log.
I20260929 11:30:31.203600 7302 log.cc:824] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:31.207585 7302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: No bootstrap required, opened a new log
I20260929 11:30:31.209726 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:31.184669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33906 (local address 127.23.153.62:37051)
0929 11:30:31.185480 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:31.185492 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:31.185538 (+ 46us) server_negotiation.cc:407] Connection header received
0929 11:30:31.187552 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:31.187581 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:31.188094 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:31.188489 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:31.190441 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:31.191919 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:31.193920 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:31.194714 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:31.194863 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:31.199931 (+ 5068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:31.199945 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:31.199967 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:31.200024 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:31.204132 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:31.205238 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:31.205248 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:31.205258 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:31.205506 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:31.205913 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:31.205919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:31.205922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:31.206690 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:31.207056 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:31.207496 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 11:30:31.208005 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":349,"threads_started":1}
I20260929 11:30:31.217252 7302 raft_consensus.cc:348] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37051 } }
I20260929 11:30:31.217777 7302 raft_consensus.cc:374] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:30:31.217906 7302 raft_consensus.cc:729] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31d2ca90b2c4b07a097a1b02f45e19a, State: Initialized, Role: FOLLOWER
I20260929 11:30:31.218554 7302 consensus_queue.cc:260] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37051 } }
I20260929 11:30:31.218796 7302 raft_consensus.cc:388] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:30:31.218931 7302 raft_consensus.cc:482] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:30:31.219136 7302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:30:31.221529 7302 raft_consensus.cc:504] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37051 } }
I20260929 11:30:31.222106 7302 leader_election.cc:302] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e31d2ca90b2c4b07a097a1b02f45e19a; no voters:
I20260929 11:30:31.222736 7302 leader_election.cc:288] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:30:31.222880 7307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:30:31.223644 7307 raft_consensus.cc:686] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 1 LEADER]: Becoming Leader. State: Replica: e31d2ca90b2c4b07a097a1b02f45e19a, State: Running, Role: LEADER
I20260929 11:30:31.224248 7307 consensus_queue.cc:237] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37051 } }
I20260929 11:30:31.224993 7302 sys_catalog.cc:564] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:30:31.227324 7309 sys_catalog.cc:455] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e31d2ca90b2c4b07a097a1b02f45e19a. Latest consensus state: current_term: 1 leader_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37051 } } }
I20260929 11:30:31.227813 7309 sys_catalog.cc:458] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [sys.catalog]: This master's current role is: LEADER
I20260929 11:30:31.229182 7308 sys_catalog.cc:455] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37051 } } }
I20260929 11:30:31.229719 7308 sys_catalog.cc:458] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [sys.catalog]: This master's current role is: LEADER
I20260929 11:30:31.231545 7316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:30:31.238124 7316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:30:31.248446 7316 catalog_manager.cc:1347] Generated new cluster ID: cf153c1eb5ef45c5a0650875b380c8fa
I20260929 11:30:31.248592 7316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:30:31.287207 7316 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:30:31.289572 7316 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:30:31.298007 7316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: Generated new TSK 0
I20260929 11:30:31.298942 7316 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
I20260929 11:30:31.364594 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:37051
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:30:31.727770 7328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:31.728099 7328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:31.728220 7328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:31.737123 7328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:31.737284 7328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:31.737455 7328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:30:31.748225 7328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:37051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:31.749634 7328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:31.751045 7328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:31.765106 7333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:31.765074 7336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:31.765035 7334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:31.765934 7328 server_base.cc:1034] running on GCE node
I20260929 11:30:31.766623 7328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:31.767869 7328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:31.769122 7328 hybrid_clock.cc:648] HybridClock initialized: now 1790681431769055 us; error 65 us; skew 500 ppm
I20260929 11:30:31.769490 7328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:31 dist-test-slave-qk05 krb5kdc[7222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681431, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:31.772982 7328 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:30:31.774600 7328 webserver.cc:460] Webserver started at http://127.23.153.1:44369/ using document root <none> and password file <none>
I20260929 11:30:31.775409 7328 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:31.775528 7328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:31.775820 7328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:30:31.779461 7328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "6caa3bf148c743d58cb9fd61f2b75788"
format_stamp: "Formatted at 2026-09-29 11:30:31 on dist-test-slave-qk05"
server_key: "c7e13b43b7d610670214ef2930a0274e"
server_key_iv: "1d4f7c01a03ccfeb0c66543d7f4f0a60"
server_key_version: "encryptionkey@0"
I20260929 11:30:31.780457 7328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "6caa3bf148c743d58cb9fd61f2b75788"
format_stamp: "Formatted at 2026-09-29 11:30:31 on dist-test-slave-qk05"
server_key: "c7e13b43b7d610670214ef2930a0274e"
server_key_iv: "1d4f7c01a03ccfeb0c66543d7f4f0a60"
server_key_version: "encryptionkey@0"
I20260929 11:30:31.786442 7328 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 11:30:31.790987 7343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:31.792928 7328 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:30:31.793139 7328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "6caa3bf148c743d58cb9fd61f2b75788"
format_stamp: "Formatted at 2026-09-29 11:30:31 on dist-test-slave-qk05"
server_key: "c7e13b43b7d610670214ef2930a0274e"
server_key_iv: "1d4f7c01a03ccfeb0c66543d7f4f0a60"
server_key_version: "encryptionkey@0"
I20260929 11:30:31.793365 7328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:31.807902 7328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:31.812635 7328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:31.812964 7328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:31.814060 7328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:30:31.815627 7328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:30:31.815719 7328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:31.815828 7328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:30:31.815907 7328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:31.849570 7328 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:32797
I20260929 11:30:31.849603 7455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:32797 every 8 connection(s)
I20260929 11:30:31.851060 7328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:30:31.851399 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 7328
I20260929 11:30:31.851600 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:30:31.852391 24164 external_mini_cluster.cc:1428] Setting key edcb11699dfc3a4d283ec5031a8a0d64
Sep 29 11:30:31 dist-test-slave-qk05 krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681431, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:30:31.874540 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:31.854232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:52035 (local address 127.23.153.62:37051)
0929 11:30:31.854713 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:31.854719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:31.856047 (+ 1328us) server_negotiation.cc:407] Connection header received
0929 11:30:31.858127 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:31.858133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:31.858226 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:31.858337 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:31.860111 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:31.860829 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:31.863089 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:31.863534 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:31.863658 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:31.868167 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:31.868174 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:31.868178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:31.868220 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:31.870782 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:31.871921 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:31.871928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:31.871931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:31.872094 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:31.872661 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:31.872666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:31.872668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:31.872954 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:31.873071 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:31.874060 (+ 989us) server_negotiation.cc:299] Negotiation successful
0929 11:30:31.874256 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":246,"threads_started":1}
WARNING: no policy specified for kudu/127.23.153.2@KRBTEST.COM; defaulting to no policy
I20260929 11:30:31.876039 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:31.855000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:52035)
0929 11:30:31.855869 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:31.855919 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:31.857718 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:31.858629 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:31.858639 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:31.859265 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:30:31.859893 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:31.859912 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:31.861009 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:31.861013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:31.862838 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:31.862856 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:31.863787 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:31.863794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:31.864102 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:31.864882 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:30:31.864911 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:30:31.867831 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:30:31.870985 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:31.870997 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:31.871047 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:31.871720 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:31.872223 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:31.872228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:31.872232 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:31.872529 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:31.873086 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:30:31.873095 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:30:31.873609 (+ 514us) client_negotiation.cc:769] Sending connection context
0929 11:30:31.873935 (+ 326us) client_negotiation.cc:240] Negotiation successful
0929 11:30:31.874543 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":338,"threads_started":1}
I20260929 11:30:31.877725 7456 heartbeater.cc:347] Connected to a master server at 127.23.153.62:37051
I20260929 11:30:31.878073 7456 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.2@KRBTEST.COM" created.
I20260929 11:30:31.878925 7456 heartbeater.cc:511] Master 127.23.153.62:37051 requested a full tablet report, sending...
I20260929 11:30:31.881376 7267 ts_manager.cc:194] Registered new tserver with Master: 6caa3bf148c743d58cb9fd61f2b75788 (127.23.153.1:32797)
I20260929 11:30:31.882895 7267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:52035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
I20260929 11:30:31.906330 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:37051
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:30:32.253719 7462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:32.254081 7462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:32.254210 7462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:32.261888 7462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:32.262048 7462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:32.262199 7462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:30:32.272871 7462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:37051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:32.274296 7462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:32.275722 7462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:32.289507 7467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:32.289815 7468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:32.289851 7470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:32.290575 7462 server_base.cc:1034] running on GCE node
I20260929 11:30:32.291261 7462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:32.292323 7462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:32.293576 7462 hybrid_clock.cc:648] HybridClock initialized: now 1790681432293511 us; error 51 us; skew 500 ppm
I20260929 11:30:32.293917 7462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:32 dist-test-slave-qk05 krb5kdc[7222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681432, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:32.297803 7462 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:30:32.299665 7462 webserver.cc:460] Webserver started at http://127.23.153.2:33493/ using document root <none> and password file <none>
I20260929 11:30:32.300449 7462 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:32.300585 7462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:32.300897 7462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:30:32.304232 7462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "1647fd55afa645ce933df9f1fbe15d9b"
format_stamp: "Formatted at 2026-09-29 11:30:32 on dist-test-slave-qk05"
server_key: "a36f4b8cdbbaa37312648ae636558107"
server_key_iv: "97acf3669cc0ce421a56f6f720acb3d6"
server_key_version: "encryptionkey@0"
I20260929 11:30:32.305104 7462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "1647fd55afa645ce933df9f1fbe15d9b"
format_stamp: "Formatted at 2026-09-29 11:30:32 on dist-test-slave-qk05"
server_key: "a36f4b8cdbbaa37312648ae636558107"
server_key_iv: "97acf3669cc0ce421a56f6f720acb3d6"
server_key_version: "encryptionkey@0"
I20260929 11:30:32.311686 7462 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 11:30:32.316207 7477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:32.317737 7462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260929 11:30:32.317935 7462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "1647fd55afa645ce933df9f1fbe15d9b"
format_stamp: "Formatted at 2026-09-29 11:30:32 on dist-test-slave-qk05"
server_key: "a36f4b8cdbbaa37312648ae636558107"
server_key_iv: "97acf3669cc0ce421a56f6f720acb3d6"
server_key_version: "encryptionkey@0"
I20260929 11:30:32.318168 7462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:32.338776 7462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:32.343312 7462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:32.343636 7462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:32.344732 7462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:30:32.346257 7462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:30:32.346355 7462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:32.346510 7462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:30:32.346566 7462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:32.379238 7462 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:34211
I20260929 11:30:32.379283 7589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:34211 every 8 connection(s)
I20260929 11:30:32.380595 7462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:30:32.380980 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 7462
I20260929 11:30:32.381155 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:30:32.381681 24164 external_mini_cluster.cc:1428] Setting key 894561a6f1908959384ea0cc1c7fab2d
Sep 29 11:30:32 dist-test-slave-qk05 krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681432, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:30:32.404953 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:32.383338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:33035 (local address 127.23.153.62:37051)
0929 11:30:32.383820 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:32.383826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:32.385388 (+ 1562us) server_negotiation.cc:407] Connection header received
0929 11:30:32.387729 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:32.387735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:32.387836 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:32.387974 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:32.389833 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:32.390897 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:32.393481 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:32.394009 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:32.394129 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:32.398591 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:32.398597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:32.398600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:32.398641 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:32.401442 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:32.402425 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:32.402431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:32.402434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:32.402577 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:32.403075 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:32.403079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:32.403081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:32.403377 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:32.403520 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:32.404312 (+ 792us) server_negotiation.cc:299] Negotiation successful
0929 11:30:32.404558 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":228,"threads_started":1}
WARNING: no policy specified for kudu/127.23.153.3@KRBTEST.COM; defaulting to no policy
I20260929 11:30:32.405951 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:32.384073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:33035)
0929 11:30:32.384947 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:32.385157 (+ 210us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:32.387366 (+ 2209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:32.388309 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:32.388319 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:32.388956 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:30:32.389605 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:32.389624 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:32.391175 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:32.391182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:32.393259 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:32.393283 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:32.394300 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:32.394308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:32.394624 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:32.395460 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:30:32.395498 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:30:32.398254 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:30:32.401659 (+ 3405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:32.401668 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:32.401686 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:32.402267 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:32.402699 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:32.402704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:32.402706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:32.402916 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:32.403519 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:30:32.403529 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:30:32.403913 (+ 384us) client_negotiation.cc:769] Sending connection context
0929 11:30:32.404172 (+ 259us) client_negotiation.cc:240] Negotiation successful
0929 11:30:32.404773 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":351,"threads_started":1}
I20260929 11:30:32.407599 7590 heartbeater.cc:347] Connected to a master server at 127.23.153.62:37051
I20260929 11:30:32.407917 7590 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.3@KRBTEST.COM" created.
I20260929 11:30:32.408831 7590 heartbeater.cc:511] Master 127.23.153.62:37051 requested a full tablet report, sending...
I20260929 11:30:32.410719 7267 ts_manager.cc:194] Registered new tserver with Master: 1647fd55afa645ce933df9f1fbe15d9b (127.23.153.2:34211)
I20260929 11:30:32.411881 7267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:33035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
I20260929 11:30:32.439152 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:37051
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:30:32.784888 7596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:32.785224 7596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:32.785329 7596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:32.793238 7596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:32.793382 7596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:32.793514 7596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:30:32.804400 7596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:37051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:32.805742 7596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:32.808120 7596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:32.823475 7601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:32.823630 7602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:32.823432 7604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:32.825047 7596 server_base.cc:1034] running on GCE node
I20260929 11:30:32.826134 7596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:32.827484 7596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:32.828751 7596 hybrid_clock.cc:648] HybridClock initialized: now 1790681432828706 us; error 48 us; skew 500 ppm
I20260929 11:30:32.829200 7596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:32 dist-test-slave-qk05 krb5kdc[7222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681432, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:32.832890 7596 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:30:32.834868 7596 webserver.cc:460] Webserver started at http://127.23.153.3:37113/ using document root <none> and password file <none>
I20260929 11:30:32.835687 7596 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:32.835829 7596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:32.836159 7596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:30:32.839632 7596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "ee8ebbd928654b818d22b6bef42dd74c"
format_stamp: "Formatted at 2026-09-29 11:30:32 on dist-test-slave-qk05"
server_key: "d1cfe413b86861c5b7109e5468d4efa1"
server_key_iv: "7972e6d9b790071eb4db6a844c032679"
server_key_version: "encryptionkey@0"
I20260929 11:30:32.840513 7596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "ee8ebbd928654b818d22b6bef42dd74c"
format_stamp: "Formatted at 2026-09-29 11:30:32 on dist-test-slave-qk05"
server_key: "d1cfe413b86861c5b7109e5468d4efa1"
server_key_iv: "7972e6d9b790071eb4db6a844c032679"
server_key_version: "encryptionkey@0"
I20260929 11:30:32.848098 7596 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260929 11:30:32.853479 7611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:32.855425 7596 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 11:30:32.855651 7596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "ee8ebbd928654b818d22b6bef42dd74c"
format_stamp: "Formatted at 2026-09-29 11:30:32 on dist-test-slave-qk05"
server_key: "d1cfe413b86861c5b7109e5468d4efa1"
server_key_iv: "7972e6d9b790071eb4db6a844c032679"
server_key_version: "encryptionkey@0"
I20260929 11:30:32.855875 7596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:32.880309 7596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:32.886238 7596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:32.886579 7596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:32.886664 7456 heartbeater.cc:503] Master 127.23.153.62:37051 was elected leader, sending a full tablet report...
I20260929 11:30:32.887830 7596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:30:32.889518 7596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:30:32.889600 7596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:32.889688 7596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:30:32.889730 7596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:32.922932 7596 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:42541
I20260929 11:30:32.922997 7723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:42541 every 8 connection(s)
I20260929 11:30:32.924770 7596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:30:32.934695 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 7596
I20260929 11:30:32.934876 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:30:32.935475 24164 external_mini_cluster.cc:1428] Setting key fbe5ce3992424bef9d3ab47e42fec58b
Sep 29 11:30:32 dist-test-slave-qk05 krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681432, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:30:32.947681 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:32.928098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:40451 (local address 127.23.153.62:37051)
0929 11:30:32.928559 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:32.928564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:32.929720 (+ 1156us) server_negotiation.cc:407] Connection header received
0929 11:30:32.932053 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:32.932058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:32.932166 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:32.932265 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:32.933971 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:32.934750 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:32.937026 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:32.937431 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:32.937556 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:32.941552 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:32.941558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:32.941561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:32.941600 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:32.944227 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:32.945152 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:32.945157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:32.945159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:32.945305 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:32.945746 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:32.945750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:32.945752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:32.946170 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:32.946315 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:32.947182 (+ 867us) server_negotiation.cc:299] Negotiation successful
0929 11:30:32.947383 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":222,"threads_started":1}
I20260929 11:30:32.948700 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:32.928582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:40451)
0929 11:30:32.929535 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:32.929588 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:32.931710 (+ 2122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:32.932548 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:32.932557 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:32.933136 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:30:32.933758 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:32.933777 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:32.934978 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:32.934986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:32.936870 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:32.936886 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:32.937683 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:32.937691 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:32.937946 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:32.938698 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:30:32.938742 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:30:32.941275 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:30:32.944427 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:32.944436 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:32.944452 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:32.944975 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:32.945410 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:32.945413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:32.945416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:32.945622 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:32.946305 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:30:32.946312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:30:32.946705 (+ 393us) client_negotiation.cc:769] Sending connection context
0929 11:30:32.946967 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 11:30:32.947469 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":434,"threads_started":1}
I20260929 11:30:32.950350 7724 heartbeater.cc:347] Connected to a master server at 127.23.153.62:37051
I20260929 11:30:32.950700 7724 heartbeater.cc:464] Registering TS with master...
I20260929 11:30:32.951630 7724 heartbeater.cc:511] Master 127.23.153.62:37051 requested a full tablet report, sending...
I20260929 11:30:32.953289 7266 ts_manager.cc:194] Registered new tserver with Master: ee8ebbd928654b818d22b6bef42dd74c (127.23.153.3:42541)
I20260929 11:30:32.954238 7266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:40451
I20260929 11:30:32.964241 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:30:32.982592 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:32.968745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33914 (local address 127.23.153.62:37051)
0929 11:30:32.968986 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:32.968992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:32.969273 (+ 281us) server_negotiation.cc:407] Connection header received
0929 11:30:32.969414 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:32.969418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:32.969513 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:32.969601 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:32.971139 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:32.971878 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:32.973447 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:32.973870 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:32.974000 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:32.976541 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:32.976547 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:32.976550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:32.976592 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:32.979299 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:32.980081 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:32.980085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:32.980088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:32.980234 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:32.980648 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:32.980652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:32.980654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:32.981032 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:32.981263 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:32.981775 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 11:30:32.982251 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 11:30:32.987416 7266 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:30:32.996519 7266 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:30:33.030337 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:33.009710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:42541 (local address 127.0.0.1:43732)
0929 11:30:33.021156 (+ 11446us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:33.021174 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:33.021303 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:33.022045 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:33.022050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:33.022091 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:33.022570 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:33.022583 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.024736 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.024741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:33.026862 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:33.026879 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.029515 (+ 2636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.029521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:33.029694 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:33.029724 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 11:30:33.029795 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 11:30:33.029914 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11320,"mutex_wait_us":98,"thread_start_us":11122,"threads_started":1}
I20260929 11:30:33.030879 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:33.009725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42906 (local address 127.23.153.1:32797)
0929 11:30:33.019940 (+ 10215us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:33.019951 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:33.020913 (+ 962us) server_negotiation.cc:407] Connection header received
0929 11:30:33.021026 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:33.021033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:33.021321 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:33.022407 (+ 1086us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:33.022686 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.024431 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.027251 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.028805 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.028994 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:33.029230 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:33.030344 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0929 11:30:33.030570 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10026,"thread_start_us":9871,"threads_started":1}
I20260929 11:30:33.030414 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:33.009585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:32797 (local address 127.0.0.1:42906)
0929 11:30:33.020397 (+ 10812us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:33.020438 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:33.020660 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:33.021566 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:33.021574 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:33.021618 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:33.022172 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:33.022194 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.024588 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.024592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:33.027053 (+ 2461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:33.027071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.029115 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.029122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:33.029345 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:33.029401 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 11:30:33.029563 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 11:30:33.029794 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10561,"mutex_wait_us":119,"thread_start_us":10395,"threads_started":1}
I20260929 11:30:33.031766 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:33.013414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.23.153.2:34211)
0929 11:30:33.021841 (+ 8427us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:33.021849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:33.022506 (+ 657us) server_negotiation.cc:407] Connection header received
0929 11:30:33.022686 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:33.022695 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:33.023385 (+ 690us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:33.023955 (+ 570us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:33.024812 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.026392 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.029119 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.030420 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.030578 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:33.030791 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:33.031378 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 11:30:33.031526 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8268,"thread_start_us":8118,"threads_started":1}
I20260929 11:30:33.031862 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:33.010266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.23.153.3:42541)
0929 11:30:33.011234 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:33.011246 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:33.021318 (+ 10072us) server_negotiation.cc:407] Connection header received
0929 11:30:33.021468 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:33.021476 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:33.021703 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:33.022709 (+ 1006us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:33.023071 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.024599 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.027116 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.028413 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.028566 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:33.031219 (+ 2653us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:33.031325 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 11:30:33.031535 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":499,"threads_started":1}
I20260929 11:30:33.032868 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:33.012493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:34211 (local address 127.0.0.1:53950)
0929 11:30:33.022361 (+ 9868us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:33.022378 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:33.022518 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:33.024214 (+ 1696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:33.024218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:33.024239 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:33.024531 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:33.024543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.026584 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.026591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:33.028882 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:33.028905 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.030634 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.030640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:33.030821 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:33.030854 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 11:30:33.030966 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 11:30:33.032522 (+ 1556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9713,"thread_start_us":9531,"threads_started":1}
I20260929 11:30:33.035497 7659 tablet_service.cc:1467] Processing CreateTablet for tablet c4cc209ea7df45778994f6bdc1cddce5 (DEFAULT_TABLE table=test-table [id=1720588dd49c4620a537ce3da3f38e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:30:33.035496 7391 tablet_service.cc:1467] Processing CreateTablet for tablet c4cc209ea7df45778994f6bdc1cddce5 (DEFAULT_TABLE table=test-table [id=1720588dd49c4620a537ce3da3f38e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:30:33.036085 7525 tablet_service.cc:1467] Processing CreateTablet for tablet c4cc209ea7df45778994f6bdc1cddce5 (DEFAULT_TABLE table=test-table [id=1720588dd49c4620a537ce3da3f38e67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:30:33.037173 7659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4cc209ea7df45778994f6bdc1cddce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:30:33.037174 7391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4cc209ea7df45778994f6bdc1cddce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:30:33.037254 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4cc209ea7df45778994f6bdc1cddce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:30:33.050228 7743 tablet_bootstrap.cc:492] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b: Bootstrap starting.
I20260929 11:30:33.052986 7744 tablet_bootstrap.cc:492] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788: Bootstrap starting.
I20260929 11:30:33.054381 7743 tablet_bootstrap.cc:654] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b: Neither blocks nor log segments found. Creating new log.
I20260929 11:30:33.055840 7745 tablet_bootstrap.cc:492] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c: Bootstrap starting.
I20260929 11:30:33.056958 7744 tablet_bootstrap.cc:654] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788: Neither blocks nor log segments found. Creating new log.
I20260929 11:30:33.057526 7743 log.cc:824] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:33.058390 7744 log.cc:824] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:33.059445 7745 tablet_bootstrap.cc:654] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c: Neither blocks nor log segments found. Creating new log.
I20260929 11:30:33.060529 7745 log.cc:824] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:33.060911 7743 tablet_bootstrap.cc:492] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b: No bootstrap required, opened a new log
I20260929 11:30:33.061213 7743 ts_tablet_manager.cc:1397] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 11:30:33.063498 7745 tablet_bootstrap.cc:492] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c: No bootstrap required, opened a new log
I20260929 11:30:33.063800 7745 ts_tablet_manager.cc:1397] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 11:30:33.066286 7744 tablet_bootstrap.cc:492] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788: No bootstrap required, opened a new log
I20260929 11:30:33.066912 7744 ts_tablet_manager.cc:1397] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20260929 11:30:33.071906 7745 raft_consensus.cc:348] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } } peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } }
I20260929 11:30:33.072352 7745 raft_consensus.cc:374] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:30:33.072444 7745 raft_consensus.cc:729] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee8ebbd928654b818d22b6bef42dd74c, State: Initialized, Role: FOLLOWER
I20260929 11:30:33.073058 7745 consensus_queue.cc:260] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } } peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } }
I20260929 11:30:33.073024 7743 raft_consensus.cc:348] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } } peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } }
I20260929 11:30:33.073499 7743 raft_consensus.cc:374] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:30:33.073613 7743 raft_consensus.cc:729] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1647fd55afa645ce933df9f1fbe15d9b, State: Initialized, Role: FOLLOWER
I20260929 11:30:33.074360 7724 heartbeater.cc:503] Master 127.23.153.62:37051 was elected leader, sending a full tablet report...
I20260929 11:30:33.074913 7745 ts_tablet_manager.cc:1428] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260929 11:30:33.075393 7743 consensus_queue.cc:260] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } } peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } }
I20260929 11:30:33.076589 7590 heartbeater.cc:503] Master 127.23.153.62:37051 was elected leader, sending a full tablet report...
I20260929 11:30:33.077340 7743 ts_tablet_manager.cc:1428] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260929 11:30:33.078294 7744 raft_consensus.cc:348] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } } peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } }
I20260929 11:30:33.078851 7744 raft_consensus.cc:374] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:30:33.078962 7744 raft_consensus.cc:729] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6caa3bf148c743d58cb9fd61f2b75788, State: Initialized, Role: FOLLOWER
I20260929 11:30:33.079660 7744 consensus_queue.cc:260] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } } peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } }
I20260929 11:30:33.083429 7744 ts_tablet_manager.cc:1428] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260929 11:30:33.092904 7749 raft_consensus.cc:482] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:30:33.093253 7749 raft_consensus.cc:504] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } } peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } }
I20260929 11:30:33.095119 7749 leader_election.cc:288] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6caa3bf148c743d58cb9fd61f2b75788 (127.23.153.1:32797), 1647fd55afa645ce933df9f1fbe15d9b (127.23.153.2:34211)
I20260929 11:30:33.102867 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:33.095512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:34641 (local address 127.23.153.1:32797)
0929 11:30:33.095734 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:33.095742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:33.096412 (+ 670us) server_negotiation.cc:407] Connection header received
0929 11:30:33.096505 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:33.096510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:33.096590 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:33.096682 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:33.097836 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.098912 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.101059 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.101919 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.102118 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:33.102204 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:33.102481 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 11:30:33.102620 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 11:30:33.102892 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:33.095638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:32797 (local address 127.23.153.3:34641)
0929 11:30:33.095942 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:33.095962 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:33.096127 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:33.096769 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:33.096774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:33.096804 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:33.097617 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:33.097663 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.099140 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.099144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:33.100770 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:33.100789 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.102118 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.102124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:33.102305 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:33.102331 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 11:30:33.102427 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 11:30:33.102583 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":48}
I20260929 11:30:33.103128 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:33.096135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:46253 (local address 127.23.153.2:34211)
0929 11:30:33.096361 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:33.096366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:33.096774 (+ 408us) server_negotiation.cc:407] Connection header received
0929 11:30:33.097045 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:33.097050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:33.097127 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:33.097224 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:33.098337 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.099294 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.101337 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.102173 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.102332 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:33.102409 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:33.102714 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 11:30:33.102851 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 11:30:33.103667 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:33.095834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:34211 (local address 127.23.153.3:46253)
0929 11:30:33.096675 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:33.096698 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:33.096874 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:33.097268 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:33.097303 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:33.097335 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:33.098164 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:33.098175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.099443 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.099449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:33.101134 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:33.101150 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.102389 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.102395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:33.102560 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:33.102584 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 11:30:33.102665 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 11:30:33.102807 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":110,"thread_start_us":551,"threads_started":1}
W20260929 11:30:33.103884 7457 tablet.cc:2369] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:30:33.104051 7411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4cc209ea7df45778994f6bdc1cddce5" candidate_uuid: "ee8ebbd928654b818d22b6bef42dd74c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6caa3bf148c743d58cb9fd61f2b75788" is_pre_election: true
I20260929 11:30:33.104638 7411 raft_consensus.cc:2454] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee8ebbd928654b818d22b6bef42dd74c in term 0.
I20260929 11:30:33.104799 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4cc209ea7df45778994f6bdc1cddce5" candidate_uuid: "ee8ebbd928654b818d22b6bef42dd74c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1647fd55afa645ce933df9f1fbe15d9b" is_pre_election: true
I20260929 11:30:33.105384 7614 leader_election.cc:302] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6caa3bf148c743d58cb9fd61f2b75788, ee8ebbd928654b818d22b6bef42dd74c; no voters:
I20260929 11:30:33.105759 7749 raft_consensus.cc:2781] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:30:33.105762 7545 raft_consensus.cc:2454] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee8ebbd928654b818d22b6bef42dd74c in term 0.
I20260929 11:30:33.105883 7749 raft_consensus.cc:482] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:30:33.105959 7749 raft_consensus.cc:3037] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:30:33.108312 7749 raft_consensus.cc:504] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } } peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } }
I20260929 11:30:33.109061 7749 leader_election.cc:288] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [CANDIDATE]: Term 1 election: Requested vote from peers 6caa3bf148c743d58cb9fd61f2b75788 (127.23.153.1:32797), 1647fd55afa645ce933df9f1fbe15d9b (127.23.153.2:34211)
I20260929 11:30:33.109596 7411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4cc209ea7df45778994f6bdc1cddce5" candidate_uuid: "ee8ebbd928654b818d22b6bef42dd74c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6caa3bf148c743d58cb9fd61f2b75788"
I20260929 11:30:33.109658 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4cc209ea7df45778994f6bdc1cddce5" candidate_uuid: "ee8ebbd928654b818d22b6bef42dd74c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1647fd55afa645ce933df9f1fbe15d9b"
I20260929 11:30:33.109803 7411 raft_consensus.cc:3037] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:30:33.109887 7545 raft_consensus.cc:3037] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:30:33.112366 7411 raft_consensus.cc:2454] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee8ebbd928654b818d22b6bef42dd74c in term 1.
I20260929 11:30:33.112460 7545 raft_consensus.cc:2454] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee8ebbd928654b818d22b6bef42dd74c in term 1.
I20260929 11:30:33.112950 7613 leader_election.cc:302] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1647fd55afa645ce933df9f1fbe15d9b, ee8ebbd928654b818d22b6bef42dd74c; no voters:
I20260929 11:30:33.113334 7749 raft_consensus.cc:2781] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:30:33.113780 7749 raft_consensus.cc:686] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [term 1 LEADER]: Becoming Leader. State: Replica: ee8ebbd928654b818d22b6bef42dd74c, State: Running, Role: LEADER
I20260929 11:30:33.114331 7749 consensus_queue.cc:237] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } } peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } }
I20260929 11:30:33.119643 7266 catalog_manager.cc:5495] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c reported cstate change: term changed from 0 to 1, leader changed from <none> to ee8ebbd928654b818d22b6bef42dd74c (127.23.153.3). New cstate: current_term: 1 leader_uuid: "ee8ebbd928654b818d22b6bef42dd74c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } health_report { overall_health: UNKNOWN } } }
W20260929 11:30:33.133286 7591 tablet.cc:2369] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:30:33.156296 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:33.150734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43748 (local address 127.23.153.3:42541)
0929 11:30:33.150985 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:33.150990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:33.151007 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:30:33.151159 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:33.151163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:33.151256 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:33.151405 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:30:33.151831 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.152567 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.154032 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.154432 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.154547 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:33.155135 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:30:33.155671 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:30:33.155810 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:33.155925 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 11:30:33.156030 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 11:30:33.166095 7545 raft_consensus.cc:1261] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b [term 1 FOLLOWER]: Refusing update from remote peer ee8ebbd928654b818d22b6bef42dd74c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:30:33.166101 7411 raft_consensus.cc:1261] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788 [term 1 FOLLOWER]: Refusing update from remote peer ee8ebbd928654b818d22b6bef42dd74c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:30:33.167312 7753 consensus_queue.cc:1035] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:30:33.167765 7749 consensus_queue.cc:1035] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 11:30:33.177807 7725 tablet.cc:2369] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:30:33.180532 7756 mvcc.cc:205] Tried to move back new op lower bound from 7334631150237327360 to 7334631150040682496. Current Snapshot: MvccSnapshot[applied={T|T < 7334631150237327360}]
I20260929 11:30:33.181277 7758 mvcc.cc:205] Tried to move back new op lower bound from 7334631150237327360 to 7334631150040682496. Current Snapshot: MvccSnapshot[applied={T|T < 7334631150237327360}]
I20260929 11:30:33.192234 7757 mvcc.cc:205] Tried to move back new op lower bound from 7334631150237327360 to 7334631150040682496. Current Snapshot: MvccSnapshot[applied={T|T < 7334631150237327360}]
I20260929 11:30:33.204051 7267 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:30:33.204370 7267 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:30:33.209635 7267 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: Sending DeleteTablet for 3 replicas of tablet c4cc209ea7df45778994f6bdc1cddce5
I20260929 11:30:33.211037 7659 tablet_service.cc:1514] Processing DeleteTablet for tablet c4cc209ea7df45778994f6bdc1cddce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:30:33 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:43732
I20260929 11:30:33.211071 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet c4cc209ea7df45778994f6bdc1cddce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:30:33 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:53950
I20260929 11:30:33.211083 7391 tablet_service.cc:1514] Processing DeleteTablet for tablet c4cc209ea7df45778994f6bdc1cddce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:30:33 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:42906
I20260929 11:30:33.211519 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 7328
I20260929 11:30:33.231544 7767 tablet_replica.cc:330] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788: stopping tablet replica
I20260929 11:30:33.232194 7767 raft_consensus.cc:2229] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:30:33.232592 7768 tablet_replica.cc:330] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b: stopping tablet replica
I20260929 11:30:33.232863 7767 raft_consensus.cc:2258] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:30:33.233222 7768 raft_consensus.cc:2229] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:30:33.233848 7768 raft_consensus.cc:2258] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:30:33.236445 7767 ts_tablet_manager.cc:1905] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:30:33.236435 7768 ts_tablet_manager.cc:1905] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:30:33.238377 7769 tablet_replica.cc:330] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c: stopping tablet replica
I20260929 11:30:33.238951 7769 raft_consensus.cc:2229] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:30:33.239743 7769 raft_consensus.cc:2258] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:30:33.242221 7769 ts_tablet_manager.cc:1905] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:30:33.244951 7768 ts_tablet_manager.cc:1918] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:30:33.245195 7768 log.cc:1196] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/c4cc209ea7df45778994f6bdc1cddce5
I20260929 11:30:33.245733 7768 ts_tablet_manager.cc:1939] T c4cc209ea7df45778994f6bdc1cddce5 P 1647fd55afa645ce933df9f1fbe15d9b: Deleting consensus metadata
I20260929 11:30:33.246353 7767 ts_tablet_manager.cc:1918] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:30:33.246681 7767 log.cc:1196] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/c4cc209ea7df45778994f6bdc1cddce5
I20260929 11:30:33.247335 7767 ts_tablet_manager.cc:1939] T c4cc209ea7df45778994f6bdc1cddce5 P 6caa3bf148c743d58cb9fd61f2b75788: Deleting consensus metadata
I20260929 11:30:33.247988 7252 catalog_manager.cc:4841] TS 1647fd55afa645ce933df9f1fbe15d9b (127.23.153.2:34211): tablet c4cc209ea7df45778994f6bdc1cddce5 (table test-table [id=1720588dd49c4620a537ce3da3f38e67]) successfully deleted
I20260929 11:30:33.249809 7253 catalog_manager.cc:4841] TS 6caa3bf148c743d58cb9fd61f2b75788 (127.23.153.1:32797): tablet c4cc209ea7df45778994f6bdc1cddce5 (table test-table [id=1720588dd49c4620a537ce3da3f38e67]) successfully deleted
I20260929 11:30:33.250254 7769 ts_tablet_manager.cc:1918] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:30:33.250468 7769 log.cc:1196] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/c4cc209ea7df45778994f6bdc1cddce5
I20260929 11:30:33.251122 7769 ts_tablet_manager.cc:1939] T c4cc209ea7df45778994f6bdc1cddce5 P ee8ebbd928654b818d22b6bef42dd74c: Deleting consensus metadata
Sep 29 11:30:33 dist-test-slave-qk05 krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:30:33.253188 7254 catalog_manager.cc:4841] TS ee8ebbd928654b818d22b6bef42dd74c (127.23.153.3:42541): tablet c4cc209ea7df45778994f6bdc1cddce5 (table test-table [id=1720588dd49c4620a537ce3da3f38e67]) successfully deleted
I20260929 11:30:33.261085 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:33.212558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42908 (local address 127.23.153.1:32797)
0929 11:30:33.212758 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:33.212765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:33.230942 (+ 18177us) server_negotiation.cc:407] Connection header received
0929 11:30:33.231512 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:33.231522 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:33.231622 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:33.231700 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:33.234351 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.235478 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.237771 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.238505 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.238622 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:33.253513 (+ 14891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:33.253537 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:33.253558 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:33.253614 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:33.257615 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:33.258516 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:33.258528 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:33.258531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:33.258720 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:33.259234 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:33.259240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:33.259244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:33.259773 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:33.260144 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:33.260426 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 11:30:33.260684 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:30:33.261982 7451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:33.356853 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 7328
I20260929 11:30:33.375087 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 7462
Sep 29 11:30:33 dist-test-slave-qk05 krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.2@KRBTEST.COM
I20260929 11:30:33.392181 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:33.375910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.23.153.2:34211)
0929 11:30:33.376089 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:33.376095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:33.376156 (+ 61us) server_negotiation.cc:407] Connection header received
0929 11:30:33.376362 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:33.376367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:33.376450 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:33.376537 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:33.378066 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.378943 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.380387 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.380845 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.381071 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:33.385638 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:33.385652 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:33.385667 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:33.385718 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:33.388913 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:33.389756 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:33.389764 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:33.389767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:33.389921 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:33.390305 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:33.390309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:33.390311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:33.390763 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:33.391103 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:33.391440 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 11:30:33.391680 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:30:33.392879 7585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:33.486857 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 7462
I20260929 11:30:33.503265 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 7596
Sep 29 11:30:33 dist-test-slave-qk05 krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.3@KRBTEST.COM
I20260929 11:30:33.522470 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:33.504248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43764 (local address 127.23.153.3:42541)
0929 11:30:33.504469 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:33.504476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:33.504500 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:30:33.504571 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:33.504576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:33.504657 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:33.504803 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:33.506152 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.507590 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.509779 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:33.510500 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:33.510699 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:33.515722 (+ 5023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:33.515744 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:33.515760 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:33.515806 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:33.519102 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:33.520177 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:33.520185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:33.520188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:33.520371 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:33.520845 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:33.520849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:33.520851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:33.521278 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:33.521554 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:33.521924 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 11:30:33.522121 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 11:30:33.523465 7719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:33.653805 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 7596
I20260929 11:30:33.671442 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 7235
I20260929 11:30:33.672639 7297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:33.792016 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 7235
I20260929 11:30:53.812813 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:37051
--webserver_interface=127.23.153.62
--webserver_port=41413
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:37051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:30:54.179656 7775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:54.180011 7775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:54.180142 7775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:54.189154 7775 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 11:30:54.189303 7775 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 11:30:54.189363 7775 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 11:30:54.189438 7775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:30:54.189513 7775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:54.189617 7775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:54.189677 7775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:30:54.189754 7775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:30:54.200721 7775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:37051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:37051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=41413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:54.202135 7775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:54.203612 7775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:54.231678 7781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:54.231632 7784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:54.231608 7782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:54.232750 7775 server_base.cc:1034] running on GCE node
I20260929 11:30:54.233700 7775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:54.235533 7775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:54.236850 7775 hybrid_clock.cc:648] HybridClock initialized: now 1790681454236740 us; error 56 us; skew 500 ppm
I20260929 11:30:54.237306 7775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:54 dist-test-slave-qk05 krb5kdc[7222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681454, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:54.245488 7775 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:30:54.247614 7775 webserver.cc:460] Webserver started at http://127.23.153.62:41413/ using document root <none> and password file <none>
I20260929 11:30:54.248503 7775 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:54.248644 7775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:54.255872 7775 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 11:30:54.260907 7791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:54.262625 7775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:30:54.262840 7775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "e31d2ca90b2c4b07a097a1b02f45e19a"
format_stamp: "Formatted at 2026-09-29 11:30:31 on dist-test-slave-qk05"
server_key: "f47b31f89441aa9350e8be5efeaf7db3"
server_key_iv: "a4447f83ee612b943d76eaf18913544e"
server_key_version: "encryptionkey@0"
I20260929 11:30:54.264050 7775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:54.287159 7775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:54.292407 7775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:54.292750 7775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:54.313285 7775 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:37051
I20260929 11:30:54.313434 7842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:37051 every 8 connection(s)
I20260929 11:30:54.315130 7775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:30:54.320129 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 7775
I20260929 11:30:54.321153 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:32797
--local_ip_for_outbound_sockets=127.23.153.1
--tserver_master_addrs=127.23.153.62:37051
--webserver_port=44369
--webserver_interface=127.23.153.1
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:30:54.325233 7843 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:30:54.331447 7843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: Bootstrap starting.
I20260929 11:30:54.353236 7843 log.cc:824] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:54.367764 7843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:30:54.368698 7843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: Bootstrap complete.
I20260929 11:30:54.379374 7843 raft_consensus.cc:348] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37051 } }
I20260929 11:30:54.380384 7843 raft_consensus.cc:729] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31d2ca90b2c4b07a097a1b02f45e19a, State: Initialized, Role: FOLLOWER
I20260929 11:30:54.381088 7843 consensus_queue.cc:260] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37051 } }
I20260929 11:30:54.381372 7843 raft_consensus.cc:388] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:30:54.381541 7843 raft_consensus.cc:482] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:30:54.381716 7843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 1 FOLLOWER]: Advancing to term 2
I20260929 11:30:54.387503 7843 raft_consensus.cc:504] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37051 } }
I20260929 11:30:54.388183 7843 leader_election.cc:302] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e31d2ca90b2c4b07a097a1b02f45e19a; no voters:
I20260929 11:30:54.388860 7843 leader_election.cc:288] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 11:30:54.389010 7847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 2 FOLLOWER]: Leader election won for term 2
I20260929 11:30:54.390504 7847 raft_consensus.cc:686] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 2 LEADER]: Becoming Leader. State: Replica: e31d2ca90b2c4b07a097a1b02f45e19a, State: Running, Role: LEADER
I20260929 11:30:54.391364 7847 consensus_queue.cc:237] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37051 } }
I20260929 11:30:54.392050 7843 sys_catalog.cc:564] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:30:54.394516 7849 sys_catalog.cc:455] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e31d2ca90b2c4b07a097a1b02f45e19a. Latest consensus state: current_term: 2 leader_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37051 } } }
I20260929 11:30:54.394506 7848 sys_catalog.cc:455] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37051 } } }
I20260929 11:30:54.395044 7848 sys_catalog.cc:458] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [sys.catalog]: This master's current role is: LEADER
I20260929 11:30:54.395040 7849 sys_catalog.cc:458] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [sys.catalog]: This master's current role is: LEADER
I20260929 11:30:54.407361 7854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:30:54.416311 7854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:30:54.418025 7854 catalog_manager.cc:1259] Loaded cluster ID: cf153c1eb5ef45c5a0650875b380c8fa
I20260929 11:30:54.418246 7854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:30:54.424212 7854 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:30:54.426913 7854 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: Loaded TSK: 0
I20260929 11:30:54.471985 7854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: Generated new TSK 1
I20260929 11:30:54.472745 7854 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 11:30:54.731441 7845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:54.731791 7845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:54.731933 7845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:54.740398 7845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:54.740551 7845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:54.740707 7845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:30:54.751636 7845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:32797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=44369
--tserver_master_addrs=127.23.153.62:37051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:54.753089 7845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:54.754508 7845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:54.772403 7871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:54.773007 7870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:54.773475 7873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:54.773658 7845 server_base.cc:1034] running on GCE node
I20260929 11:30:54.774732 7845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:54.776528 7845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:54.777834 7845 hybrid_clock.cc:648] HybridClock initialized: now 1790681454777760 us; error 67 us; skew 500 ppm
I20260929 11:30:54.778262 7845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:54 dist-test-slave-qk05 krb5kdc[7222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681454, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:54.782202 7845 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:30:54.784077 7845 webserver.cc:460] Webserver started at http://127.23.153.1:44369/ using document root <none> and password file <none>
I20260929 11:30:54.784870 7845 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:54.785027 7845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:54.791320 7845 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:30:54.795775 7880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:54.797900 7845 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:30:54.798295 7845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "6caa3bf148c743d58cb9fd61f2b75788"
format_stamp: "Formatted at 2026-09-29 11:30:31 on dist-test-slave-qk05"
server_key: "c7e13b43b7d610670214ef2930a0274e"
server_key_iv: "1d4f7c01a03ccfeb0c66543d7f4f0a60"
server_key_version: "encryptionkey@0"
I20260929 11:30:54.800021 7845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:54.815150 7845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:54.820969 7845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:54.821565 7845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:54.823055 7845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:30:54.825212 7845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:30:54.825337 7845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:54.825466 7845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:30:54.825542 7845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:54.862329 7845 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:32797
I20260929 11:30:54.862385 7992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:32797 every 8 connection(s)
I20260929 11:30:54.863883 7845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:30:54.872278 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 7845
I20260929 11:30:54.873096 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:34211
--local_ip_for_outbound_sockets=127.23.153.2
--tserver_master_addrs=127.23.153.62:37051
--webserver_port=33493
--webserver_interface=127.23.153.2
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:30:54 dist-test-slave-qk05 krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681454, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:30:54.891556 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:54.867184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:56151)
0929 11:30:54.868028 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:54.868083 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:54.869952 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:54.871630 (+ 1678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:54.871640 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:54.872447 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:30:54.873307 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:54.873338 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:54.875334 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:54.875340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:54.877203 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:54.877221 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:54.878440 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:54.878447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:54.878709 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:54.879489 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:30:54.879520 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:30:54.882422 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:30:54.886786 (+ 4364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:54.886795 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:54.886809 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:54.887418 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:54.888030 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:54.888034 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:54.888037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:54.888261 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:54.889256 (+ 995us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:30:54.889264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:30:54.889608 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 11:30:54.889864 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 11:30:54.890357 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":330,"threads_started":1}
I20260929 11:30:54.891844 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:54.866870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:56151 (local address 127.23.153.62:37051)
0929 11:30:54.867478 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:54.867492 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:54.868241 (+ 749us) server_negotiation.cc:407] Connection header received
0929 11:30:54.870571 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:54.870586 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:54.870996 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:54.871497 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:54.873620 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:54.874954 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:54.877456 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:54.878180 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:54.878346 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:54.882792 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:54.882802 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:54.882820 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:54.882868 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:54.886538 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:54.887581 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:54.887590 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:54.887599 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:54.887854 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:54.888397 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:54.888401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:54.888404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:54.889072 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:54.889259 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:54.890147 (+ 888us) server_negotiation.cc:299] Negotiation successful
0929 11:30:54.890654 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":291,"threads_started":1}
I20260929 11:30:54.895042 7993 heartbeater.cc:347] Connected to a master server at 127.23.153.62:37051
I20260929 11:30:54.895380 7993 heartbeater.cc:464] Registering TS with master...
I20260929 11:30:54.896111 7993 heartbeater.cc:511] Master 127.23.153.62:37051 requested a full tablet report, sending...
I20260929 11:30:54.898510 7808 ts_manager.cc:194] Registered new tserver with Master: 6caa3bf148c743d58cb9fd61f2b75788 (127.23.153.1:32797)
I20260929 11:30:54.900089 7808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:56151
W20260929 11:30:55.245842 7997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:55.246183 7997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:55.246316 7997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:55.254241 7997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:55.254412 7997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:55.254570 7997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:30:55.265038 7997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:34211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=33493
--tserver_master_addrs=127.23.153.62:37051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:55.266474 7997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:55.267940 7997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:55.283776 8005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:55.283860 8003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:55.283830 8002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:55.289327 7997 server_base.cc:1034] running on GCE node
I20260929 11:30:55.290261 7997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:55.291767 7997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:55.293172 7997 hybrid_clock.cc:648] HybridClock initialized: now 1790681455293091 us; error 64 us; skew 500 ppm
I20260929 11:30:55.293658 7997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:55 dist-test-slave-qk05 krb5kdc[7222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681455, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:55.297798 7997 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:30:55.299923 7997 webserver.cc:460] Webserver started at http://127.23.153.2:33493/ using document root <none> and password file <none>
I20260929 11:30:55.300773 7997 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:55.300952 7997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:55.308429 7997 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:30:55.311802 8012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:55.313845 7997 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:30:55.314103 7997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "1647fd55afa645ce933df9f1fbe15d9b"
format_stamp: "Formatted at 2026-09-29 11:30:32 on dist-test-slave-qk05"
server_key: "a36f4b8cdbbaa37312648ae636558107"
server_key_iv: "97acf3669cc0ce421a56f6f720acb3d6"
server_key_version: "encryptionkey@0"
I20260929 11:30:55.315574 7997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:55.333812 7997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:55.338872 7997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:55.339214 7997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:55.340281 7997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:30:55.341745 7997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:30:55.341836 7997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:55.341919 7997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:30:55.341971 7997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:55.375851 7997 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:34211
I20260929 11:30:55.375936 8124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:34211 every 8 connection(s)
I20260929 11:30:55.377547 7997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:30:55.382969 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 7997
I20260929 11:30:55.383843 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:42541
--local_ip_for_outbound_sockets=127.23.153.3
--tserver_master_addrs=127.23.153.62:37051
--webserver_port=37113
--webserver_interface=127.23.153.3
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 11:30:55 dist-test-slave-qk05 krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681455, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:30:55.402552 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:55.380623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:60869 (local address 127.23.153.62:37051)
0929 11:30:55.380813 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:55.380818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:55.382215 (+ 1397us) server_negotiation.cc:407] Connection header received
0929 11:30:55.384980 (+ 2765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:55.385006 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:55.385140 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:55.385240 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:55.387544 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:55.388423 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:55.390862 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:55.391452 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:55.391586 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:55.395840 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:55.395846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:55.395849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:55.395890 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:55.398811 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:55.399818 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:55.399823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:55.399825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:55.399973 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:55.400563 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:55.400567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:55.400570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:55.400893 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:55.401051 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:55.401908 (+ 857us) server_negotiation.cc:299] Negotiation successful
0929 11:30:55.402191 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 11:30:55.403672 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:55.381161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:60869)
0929 11:30:55.381984 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:55.382060 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:55.384580 (+ 2520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:55.385569 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:55.385583 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:55.386351 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:30:55.387283 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:55.387311 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:55.388611 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:55.388617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:55.390588 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:55.390611 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:55.391730 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:55.391738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:55.392037 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:55.392891 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:30:55.392919 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:30:55.395528 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:30:55.399102 (+ 3574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:55.399112 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:55.399129 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:55.399653 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:55.400167 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:55.400173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:55.400177 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:55.400418 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:55.401052 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:30:55.401063 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:30:55.401480 (+ 417us) client_negotiation.cc:769] Sending connection context
0929 11:30:55.401761 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 11:30:55.402388 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":340,"threads_started":1}
I20260929 11:30:55.405637 8125 heartbeater.cc:347] Connected to a master server at 127.23.153.62:37051
I20260929 11:30:55.406097 8125 heartbeater.cc:464] Registering TS with master...
I20260929 11:30:55.406996 8125 heartbeater.cc:511] Master 127.23.153.62:37051 requested a full tablet report, sending...
I20260929 11:30:55.409068 7808 ts_manager.cc:194] Registered new tserver with Master: 1647fd55afa645ce933df9f1fbe15d9b (127.23.153.2:34211)
I20260929 11:30:55.410311 7808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:60869
W20260929 11:30:55.752267 8128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:30:55.752586 8128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:30:55.752686 8128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:30:55.760181 8128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:30:55.760317 8128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:30:55.760435 8128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:30:55.771000 8128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:42541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=37113
--tserver_master_addrs=127.23.153.62:37051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:30:55.772373 8128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:30:55.773703 8128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:30:55.788196 8133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:55.788385 8134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:30:55.788762 8136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:30:55.789384 8128 server_base.cc:1034] running on GCE node
I20260929 11:30:55.790059 8128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:30:55.791203 8128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:30:55.792454 8128 hybrid_clock.cc:648] HybridClock initialized: now 1790681455792381 us; error 58 us; skew 500 ppm
I20260929 11:30:55.792838 8128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:30:55 dist-test-slave-qk05 krb5kdc[7222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681455, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:30:55.797423 8128 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:30:55.799759 8128 webserver.cc:460] Webserver started at http://127.23.153.3:37113/ using document root <none> and password file <none>
I20260929 11:30:55.800704 8128 fs_manager.cc:359] Metadata directory not provided
I20260929 11:30:55.800894 8128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:30:55.808888 8128 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 11:30:55.812424 8143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:55.814196 8128 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:30:55.814396 8128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "ee8ebbd928654b818d22b6bef42dd74c"
format_stamp: "Formatted at 2026-09-29 11:30:32 on dist-test-slave-qk05"
server_key: "d1cfe413b86861c5b7109e5468d4efa1"
server_key_iv: "7972e6d9b790071eb4db6a844c032679"
server_key_version: "encryptionkey@0"
I20260929 11:30:55.815840 8128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:30:55.840090 8128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:30:55.845702 8128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:30:55.845999 8128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:30:55.847780 8128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:30:55.850101 8128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:30:55.850253 8128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:55.850390 8128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:30:55.850476 8128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:30:55.885634 8128 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:42541
I20260929 11:30:55.885681 8255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:42541 every 8 connection(s)
I20260929 11:30:55.887322 8128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:30:55.893090 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 8128
Sep 29 11:30:55 dist-test-slave-qk05 krb5kdc[7222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681455, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:30:55.903738 7993 heartbeater.cc:503] Master 127.23.153.62:37051 was elected leader, sending a full tablet report...
I20260929 11:30:55.911908 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:55.893992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41754 (local address 127.23.153.62:37051)
0929 11:30:55.894471 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:55.894477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:55.894793 (+ 316us) server_negotiation.cc:407] Connection header received
0929 11:30:55.894934 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:55.894939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:55.895008 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:55.895239 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:55.896806 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:55.897667 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:55.899280 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:55.899942 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:55.900066 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:55.903528 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:55.903535 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:55.903539 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:55.903590 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:55.908352 (+ 4762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:55.909369 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:55.909374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:55.909376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:55.909543 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:55.909995 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:55.910001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:55.910004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:55.910445 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:55.910711 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:55.911109 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 11:30:55.911389 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":363,"spinlock_wait_cycles":2944,"thread_start_us":242,"threads_started":1}
I20260929 11:30:55.913174 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:55.889957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:53053 (local address 127.23.153.62:37051)
0929 11:30:55.890152 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:55.890157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:55.891555 (+ 1398us) server_negotiation.cc:407] Connection header received
0929 11:30:55.893583 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:55.893589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:55.893678 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:55.893768 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:55.895852 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:55.896597 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:55.898286 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:55.898707 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:55.898821 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:55.904310 (+ 5489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:55.904317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:55.904321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:55.904384 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:55.908327 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:55.910082 (+ 1755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:55.910086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:55.910089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:55.910228 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:55.910722 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:55.910725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:55.910728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:55.911059 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:55.911382 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:55.912219 (+ 837us) server_negotiation.cc:299] Negotiation successful
0929 11:30:55.912714 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"spinlock_wait_cycles":15616}
I20260929 11:30:55.914434 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:55.890467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:53053)
0929 11:30:55.891368 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:55.891418 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:55.893237 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:55.894042 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:55.894053 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:55.894744 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:30:55.895672 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:55.895694 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:55.896774 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:55.896778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:55.898128 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:55.898143 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:55.898963 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:55.898970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:55.899474 (+ 504us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:55.900786 (+ 1312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:30:55.900822 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:30:55.904021 (+ 3199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:30:55.909400 (+ 5379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:55.909410 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:55.909427 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:55.909941 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:55.910344 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:30:55.910348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:30:55.910350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:30:55.910568 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:30:55.911329 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:30:55.911340 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:30:55.911746 (+ 406us) client_negotiation.cc:769] Sending connection context
0929 11:30:55.912011 (+ 265us) client_negotiation.cc:240] Negotiation successful
0929 11:30:55.912680 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":293,"threads_started":1}
I20260929 11:30:55.916985 8256 heartbeater.cc:347] Connected to a master server at 127.23.153.62:37051
I20260929 11:30:55.917425 8256 heartbeater.cc:464] Registering TS with master...
I20260929 11:30:55.918242 8256 heartbeater.cc:511] Master 127.23.153.62:37051 requested a full tablet report, sending...
I20260929 11:30:55.920073 7808 ts_manager.cc:194] Registered new tserver with Master: ee8ebbd928654b818d22b6bef42dd74c (127.23.153.3:42541)
I20260929 11:30:55.921002 7808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:53053
I20260929 11:30:55.928155 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:30:55.935635 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:55.929289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41760 (local address 127.23.153.62:37051)
0929 11:30:55.929543 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:55.929551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:55.929930 (+ 379us) server_negotiation.cc:407] Connection header received
0929 11:30:55.930194 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:55.930198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:55.930365 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:55.930518 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:30:55.930968 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:55.931848 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:55.933595 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:55.934032 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:55.934150 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:55.934697 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:30:55.934933 (+ 236us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 11:30:55.935363 (+ 430us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41760: token expired
Metrics: {"server-negotiator.queue_time_us":93}
W20260929 11:30:55.935758 7995 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41760: token expired
W20260929 11:30:55.936623 7731 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.153.62:37051)
I20260929 11:30:55.951414 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:55.937203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.23.153.62:37051)
0929 11:30:55.937404 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:55.937409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:55.937430 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:30:55.937549 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:55.937554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:55.937625 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:55.937714 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:55.939235 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:55.940027 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:55.941847 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:55.942307 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:55.942434 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:55.945608 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:55.945614 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:55.945617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:55.945660 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:55.948356 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:55.949273 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:55.949278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:55.949281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:55.949428 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:55.949817 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:55.949821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:55.949823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:55.950182 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:55.950424 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:55.950805 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 11:30:55.951092 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 11:30:55.955406 7808 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:30:55.962921 7808 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:30:55.999146 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:55.976397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:42541 (local address 127.0.0.1:54584)
0929 11:30:55.978741 (+ 2344us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:55.978792 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:55.979045 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:55.983070 (+ 4025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:55.983082 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:55.983131 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:55.985304 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:55.985334 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:55.989926 (+ 4592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:55.989934 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:55.992926 (+ 2992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:55.992959 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:55.997851 (+ 4892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:55.997860 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:55.998161 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:55.998226 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 11:30:55.998392 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 11:30:55.998654 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2032,"threads_started":1}
I20260929 11:30:56.002439 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:55.976497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54584 (local address 127.23.153.3:42541)
0929 11:30:55.978050 (+ 1553us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:55.978062 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:55.979284 (+ 1222us) server_negotiation.cc:407] Connection header received
0929 11:30:55.980973 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:55.980986 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:55.982367 (+ 1381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:55.982636 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:55.985620 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:55.987929 (+ 2309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:55.993310 (+ 5381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:55.996904 (+ 3594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:55.997215 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:55.997555 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:55.999947 (+ 2392us) server_negotiation.cc:299] Negotiation successful
0929 11:30:56.001868 (+ 1921us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1326,"thread_start_us":1735,"threads_started":1}
I20260929 11:30:56.008622 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 4764559aed5d42b2b742eaa5b456db8e (DEFAULT_TABLE table=test-table [id=d39201f28158488fba41bd9de648ca01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:30:56.010454 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4764559aed5d42b2b742eaa5b456db8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:30:56.011513 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:55.977486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.23.153.1:32797)
0929 11:30:56.003384 (+ 25898us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:56.003395 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:56.003436 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:30:56.003749 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:56.003760 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:56.004069 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:56.004300 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:56.005007 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.006287 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.008775 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.010083 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.010224 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:56.010472 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:56.011136 (+ 664us) server_negotiation.cc:299] Negotiation successful
0929 11:30:56.011276 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25671,"thread_start_us":25952,"threads_started":1}
I20260929 11:30:56.011766 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:55.977956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:32797 (local address 127.0.0.1:55054)
0929 11:30:56.003269 (+ 25313us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:56.003293 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:56.003465 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:56.004427 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:56.004432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:56.004465 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:56.004835 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:56.004849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.006481 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.006485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:56.008546 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:56.008565 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.010309 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.010315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:56.010513 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:56.010548 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:30:56.010640 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 11:30:56.010768 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25139,"thread_start_us":24800,"threads_started":1}
I20260929 11:30:56.011840 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:55.976801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:34211 (local address 127.0.0.1:50894)
0929 11:30:55.999495 (+ 22694us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:55.999520 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:55.999676 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:56.005045 (+ 5369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:56.005050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:56.005075 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:56.005247 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:56.005255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.007274 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.007280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:56.009237 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:56.009254 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.011126 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.011132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:56.011311 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:56.011346 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:30:56.011415 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 11:30:56.011561 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22515,"thread_start_us":1693,"threads_started":1}
I20260929 11:30:56.012883 8269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:55.977369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50894 (local address 127.23.153.2:34211)
0929 11:30:56.004434 (+ 27065us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:56.004442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:56.004479 (+ 37us) server_negotiation.cc:407] Connection header received
0929 11:30:56.004574 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:56.004582 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:56.004834 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:56.005421 (+ 587us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:56.005659 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.007133 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.009417 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.010885 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.012105 (+ 1220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:56.012305 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:56.012377 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 11:30:56.012561 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26916,"thread_start_us":26783,"threads_started":1}
I20260929 11:30:56.015797 7928 tablet_service.cc:1467] Processing CreateTablet for tablet 4764559aed5d42b2b742eaa5b456db8e (DEFAULT_TABLE table=test-table [id=d39201f28158488fba41bd9de648ca01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:30:56.016656 8060 tablet_service.cc:1467] Processing CreateTablet for tablet 4764559aed5d42b2b742eaa5b456db8e (DEFAULT_TABLE table=test-table [id=d39201f28158488fba41bd9de648ca01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:30:56.017854 7928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4764559aed5d42b2b742eaa5b456db8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:30:56.018483 8060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4764559aed5d42b2b742eaa5b456db8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:30:56.024654 8272 tablet_bootstrap.cc:492] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c: Bootstrap starting.
I20260929 11:30:56.028635 8272 tablet_bootstrap.cc:654] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c: Neither blocks nor log segments found. Creating new log.
I20260929 11:30:56.030086 8272 log.cc:824] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:56.036158 8274 tablet_bootstrap.cc:492] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788: Bootstrap starting.
I20260929 11:30:56.038275 8272 tablet_bootstrap.cc:492] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c: No bootstrap required, opened a new log
I20260929 11:30:56.038604 8272 ts_tablet_manager.cc:1397] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260929 11:30:56.039830 8275 tablet_bootstrap.cc:492] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b: Bootstrap starting.
I20260929 11:30:56.040192 8274 tablet_bootstrap.cc:654] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788: Neither blocks nor log segments found. Creating new log.
I20260929 11:30:56.041540 8274 log.cc:824] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:56.043670 8275 tablet_bootstrap.cc:654] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b: Neither blocks nor log segments found. Creating new log.
I20260929 11:30:56.044859 8275 log.cc:824] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b: Log is configured to *not* fsync() on all Append() calls
I20260929 11:30:56.045905 8274 tablet_bootstrap.cc:492] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788: No bootstrap required, opened a new log
I20260929 11:30:56.046298 8274 ts_tablet_manager.cc:1397] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20260929 11:30:56.047946 8275 tablet_bootstrap.cc:492] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b: No bootstrap required, opened a new log
I20260929 11:30:56.048229 8275 ts_tablet_manager.cc:1397] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 11:30:56.049263 8272 raft_consensus.cc:348] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } } peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } }
I20260929 11:30:56.049767 8272 raft_consensus.cc:374] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:30:56.049918 8272 raft_consensus.cc:729] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee8ebbd928654b818d22b6bef42dd74c, State: Initialized, Role: FOLLOWER
I20260929 11:30:56.050663 8272 consensus_queue.cc:260] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } } peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } }
I20260929 11:30:56.051977 8256 heartbeater.cc:503] Master 127.23.153.62:37051 was elected leader, sending a full tablet report...
I20260929 11:30:56.052664 8272 ts_tablet_manager.cc:1428] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260929 11:30:56.056509 8274 raft_consensus.cc:348] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } } peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } }
I20260929 11:30:56.057014 8274 raft_consensus.cc:374] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:30:56.057125 8274 raft_consensus.cc:729] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6caa3bf148c743d58cb9fd61f2b75788, State: Initialized, Role: FOLLOWER
I20260929 11:30:56.056975 8275 raft_consensus.cc:348] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } } peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } }
I20260929 11:30:56.057519 8275 raft_consensus.cc:374] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:30:56.057621 8275 raft_consensus.cc:729] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1647fd55afa645ce933df9f1fbe15d9b, State: Initialized, Role: FOLLOWER
I20260929 11:30:56.057775 8274 consensus_queue.cc:260] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } } peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } }
I20260929 11:30:56.058374 8275 consensus_queue.cc:260] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } } peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } }
I20260929 11:30:56.060122 8125 heartbeater.cc:503] Master 127.23.153.62:37051 was elected leader, sending a full tablet report...
I20260929 11:30:56.060874 8275 ts_tablet_manager.cc:1428] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 11:30:56.060990 8274 ts_tablet_manager.cc:1428] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260929 11:30:56.101218 8279 raft_consensus.cc:482] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:30:56.101547 8279 raft_consensus.cc:504] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } } peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } }
I20260929 11:30:56.103729 8279 leader_election.cc:288] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee8ebbd928654b818d22b6bef42dd74c (127.23.153.3:42541), 1647fd55afa645ce933df9f1fbe15d9b (127.23.153.2:34211)
I20260929 11:30:56.113237 8269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:56.104291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:40561 (local address 127.23.153.2:34211)
0929 11:30:56.104526 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:56.104533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:56.105161 (+ 628us) server_negotiation.cc:407] Connection header received
0929 11:30:56.105254 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:56.105260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:56.105338 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:56.105451 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:56.106255 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.107431 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.110473 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.111791 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.112166 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:56.112259 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:56.112805 (+ 546us) server_negotiation.cc:299] Negotiation successful
0929 11:30:56.112961 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 11:30:56.113237 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:56.104197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:32933 (local address 127.23.153.3:42541)
0929 11:30:56.104712 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:56.104718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:56.105003 (+ 285us) server_negotiation.cc:407] Connection header received
0929 11:30:56.105256 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:56.105262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:56.105338 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:56.105444 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:30:56.106300 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.107478 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.110469 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.111805 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.112169 (+ 364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:56.112258 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:56.112805 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 11:30:56.112961 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366}
I20260929 11:30:56.113464 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:56.104175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:34211 (local address 127.23.153.1:40561)
0929 11:30:56.105030 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:56.105044 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:56.105155 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:56.105500 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:56.105505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:56.105538 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:56.106030 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:56.106048 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.107652 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.107657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:56.110188 (+ 2531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:56.110214 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.112300 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.112305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:56.112544 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:56.112617 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 11:30:56.112742 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 11:30:56.112938 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":229,"thread_start_us":605,"threads_started":1}
I20260929 11:30:56.113479 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:56.103955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:42541 (local address 127.23.153.1:32933)
0929 11:30:56.104856 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0929 11:30:56.104876 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:30:56.105067 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:30:56.105500 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:30:56.105505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:30:56.105612 (+ 107us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:30:56.106120 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:56.106130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.107666 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.107670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:56.110188 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:30:56.110214 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.112290 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.112298 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:30:56.112541 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:56.112627 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 11:30:56.112742 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 11:30:56.112938 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":152,"spinlock_wait_cycles":28544,"thread_start_us":501,"threads_started":1}
I20260929 11:30:56.114672 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764559aed5d42b2b742eaa5b456db8e" candidate_uuid: "6caa3bf148c743d58cb9fd61f2b75788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee8ebbd928654b818d22b6bef42dd74c" is_pre_election: true
I20260929 11:30:56.114696 8080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764559aed5d42b2b742eaa5b456db8e" candidate_uuid: "6caa3bf148c743d58cb9fd61f2b75788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1647fd55afa645ce933df9f1fbe15d9b" is_pre_election: true
I20260929 11:30:56.115218 8211 raft_consensus.cc:2454] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6caa3bf148c743d58cb9fd61f2b75788 in term 0.
I20260929 11:30:56.115217 8080 raft_consensus.cc:2454] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6caa3bf148c743d58cb9fd61f2b75788 in term 0.
I20260929 11:30:56.115967 7884 leader_election.cc:302] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1647fd55afa645ce933df9f1fbe15d9b, 6caa3bf148c743d58cb9fd61f2b75788, ee8ebbd928654b818d22b6bef42dd74c; no voters:
W20260929 11:30:56.116426 7994 tablet.cc:2369] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:30:56.116426 8279 raft_consensus.cc:2781] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:30:56.116642 8279 raft_consensus.cc:482] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:30:56.116742 8279 raft_consensus.cc:3037] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:30:56.119071 8279 raft_consensus.cc:504] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } } peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } }
I20260929 11:30:56.119807 8279 leader_election.cc:288] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [CANDIDATE]: Term 1 election: Requested vote from peers ee8ebbd928654b818d22b6bef42dd74c (127.23.153.3:42541), 1647fd55afa645ce933df9f1fbe15d9b (127.23.153.2:34211)
I20260929 11:30:56.120482 8080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764559aed5d42b2b742eaa5b456db8e" candidate_uuid: "6caa3bf148c743d58cb9fd61f2b75788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1647fd55afa645ce933df9f1fbe15d9b"
I20260929 11:30:56.120738 8080 raft_consensus.cc:3037] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:30:56.122731 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4764559aed5d42b2b742eaa5b456db8e" candidate_uuid: "6caa3bf148c743d58cb9fd61f2b75788" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee8ebbd928654b818d22b6bef42dd74c"
I20260929 11:30:56.122984 8211 raft_consensus.cc:3037] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:30:56.123320 8080 raft_consensus.cc:2454] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6caa3bf148c743d58cb9fd61f2b75788 in term 1.
I20260929 11:30:56.123900 7882 leader_election.cc:302] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1647fd55afa645ce933df9f1fbe15d9b, 6caa3bf148c743d58cb9fd61f2b75788; no voters:
I20260929 11:30:56.124300 8279 raft_consensus.cc:2781] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:30:56.124810 8279 raft_consensus.cc:686] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [term 1 LEADER]: Becoming Leader. State: Replica: 6caa3bf148c743d58cb9fd61f2b75788, State: Running, Role: LEADER
I20260929 11:30:56.125412 8279 consensus_queue.cc:237] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } } peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } }
I20260929 11:30:56.126160 8211 raft_consensus.cc:2454] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6caa3bf148c743d58cb9fd61f2b75788 in term 1.
W20260929 11:30:56.129676 8126 tablet.cc:2369] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:30:56.131644 7807 catalog_manager.cc:5495] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6caa3bf148c743d58cb9fd61f2b75788 (127.23.153.1). New cstate: current_term: 1 leader_uuid: "6caa3bf148c743d58cb9fd61f2b75788" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6caa3bf148c743d58cb9fd61f2b75788" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 32797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 } health_report { overall_health: UNKNOWN } } }
W20260929 11:30:56.139153 8257 tablet.cc:2369] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:30:56.185738 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:56.179383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55062 (local address 127.23.153.1:32797)
0929 11:30:56.179626 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:56.179632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:56.179653 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:30:56.179857 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:56.179861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:56.179958 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:56.180111 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:30:56.180480 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.181230 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.182922 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.183520 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.183637 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:56.184265 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:30:56.184810 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:30:56.184942 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:56.185027 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 11:30:56.185290 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 11:30:56.195147 8211 raft_consensus.cc:1261] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c [term 1 FOLLOWER]: Refusing update from remote peer 6caa3bf148c743d58cb9fd61f2b75788: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:30:56.195147 8080 raft_consensus.cc:1261] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b [term 1 FOLLOWER]: Refusing update from remote peer 6caa3bf148c743d58cb9fd61f2b75788: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:30:56.196601 8283 consensus_queue.cc:1035] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1647fd55afa645ce933df9f1fbe15d9b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 34211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:30:56.196985 8279 consensus_queue.cc:1035] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee8ebbd928654b818d22b6bef42dd74c" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 42541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:30:56.237716 7808 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:30:56.238106 7808 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:30:56.242802 7808 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: Sending DeleteTablet for 3 replicas of tablet 4764559aed5d42b2b742eaa5b456db8e
I20260929 11:30:56.244705 7928 tablet_service.cc:1514] Processing DeleteTablet for tablet 4764559aed5d42b2b742eaa5b456db8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:30:56 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:55054
I20260929 11:30:56.244751 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet 4764559aed5d42b2b742eaa5b456db8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:30:56 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:54584
I20260929 11:30:56.245380 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 7845
I20260929 11:30:56.254109 8060 tablet_service.cc:1514] Processing DeleteTablet for tablet 4764559aed5d42b2b742eaa5b456db8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:30:56 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:50894
I20260929 11:30:56.256783 8299 tablet_replica.cc:330] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b: stopping tablet replica
I20260929 11:30:56.257751 8298 tablet_replica.cc:330] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788: stopping tablet replica
I20260929 11:30:56.257931 8299 raft_consensus.cc:2229] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:30:56.258529 8299 raft_consensus.cc:2258] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:30:56.259083 8298 raft_consensus.cc:2229] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:30:56.259409 8297 tablet_replica.cc:330] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c: stopping tablet replica
I20260929 11:30:56.259865 8297 raft_consensus.cc:2229] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:30:56.260072 8298 raft_consensus.cc:2258] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:30:56.260419 8297 raft_consensus.cc:2258] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:30:56.262606 8298 ts_tablet_manager.cc:1905] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:30:56.262637 8299 ts_tablet_manager.cc:1905] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:30:56.269232 8297 ts_tablet_manager.cc:1905] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:30:56.270067 8299 ts_tablet_manager.cc:1918] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:30:56.270282 8299 log.cc:1196] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/4764559aed5d42b2b742eaa5b456db8e
I20260929 11:30:56.270808 8299 ts_tablet_manager.cc:1939] T 4764559aed5d42b2b742eaa5b456db8e P 1647fd55afa645ce933df9f1fbe15d9b: Deleting consensus metadata
I20260929 11:30:56.271713 8298 ts_tablet_manager.cc:1918] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:30:56.272006 8298 log.cc:1196] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/4764559aed5d42b2b742eaa5b456db8e
I20260929 11:30:56.272611 8298 ts_tablet_manager.cc:1939] T 4764559aed5d42b2b742eaa5b456db8e P 6caa3bf148c743d58cb9fd61f2b75788: Deleting consensus metadata
I20260929 11:30:56.273094 7793 catalog_manager.cc:4841] TS 1647fd55afa645ce933df9f1fbe15d9b (127.23.153.2:34211): tablet 4764559aed5d42b2b742eaa5b456db8e (table test-table [id=d39201f28158488fba41bd9de648ca01]) successfully deleted
I20260929 11:30:56.274573 7794 catalog_manager.cc:4841] TS 6caa3bf148c743d58cb9fd61f2b75788 (127.23.153.1:32797): tablet 4764559aed5d42b2b742eaa5b456db8e (table test-table [id=d39201f28158488fba41bd9de648ca01]) successfully deleted
I20260929 11:30:56.276473 8297 ts_tablet_manager.cc:1918] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:30:56.276752 8297 log.cc:1196] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/4764559aed5d42b2b742eaa5b456db8e
I20260929 11:30:56.277447 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:56.247346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.23.153.1:32797)
0929 11:30:56.247592 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:56.247599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:56.248165 (+ 566us) server_negotiation.cc:407] Connection header received
0929 11:30:56.248261 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:56.248268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:56.248369 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:56.248488 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:56.251303 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.252808 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.257377 (+ 4569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.258171 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.258330 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:56.262536 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:56.262553 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:56.262571 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:56.262636 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:56.267228 (+ 4592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:56.274796 (+ 7568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:56.274807 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:56.274811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:56.275067 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:56.275568 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:56.275574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:56.275577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:56.276077 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:56.276405 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:56.276756 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 11:30:56.277054 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260929 11:30:56.277953 8297 ts_tablet_manager.cc:1939] T 4764559aed5d42b2b742eaa5b456db8e P ee8ebbd928654b818d22b6bef42dd74c: Deleting consensus metadata
I20260929 11:30:56.278216 7988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:56.279902 7795 catalog_manager.cc:4841] TS ee8ebbd928654b818d22b6bef42dd74c (127.23.153.3:42541): tablet 4764559aed5d42b2b742eaa5b456db8e (table test-table [id=d39201f28158488fba41bd9de648ca01]) successfully deleted
I20260929 11:30:56.381743 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 7845
I20260929 11:30:56.399138 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 7997
I20260929 11:30:56.413436 8269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:56.399876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50902 (local address 127.23.153.2:34211)
0929 11:30:56.400096 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:56.400101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:56.400120 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:30:56.400215 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:56.400219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:56.400300 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:56.400379 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:56.401722 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.402488 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.403969 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.404380 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.404480 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:56.407227 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:56.407245 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:56.407263 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:56.407344 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:56.410413 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:56.411347 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:56.411354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:56.411357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:56.411497 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:56.411827 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:56.411829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:56.411831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:56.412239 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:56.412459 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:56.412794 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 11:30:56.412999 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 11:30:56.414192 8120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:56.507728 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 7997
I20260929 11:30:56.522830 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 8128
I20260929 11:30:56.540441 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:30:56.523937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54590 (local address 127.23.153.3:42541)
0929 11:30:56.524306 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0929 11:30:56.524314 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:30:56.524344 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:30:56.524481 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:30:56.524485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:30:56.524572 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:30:56.524701 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:30:56.527000 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.528051 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.529796 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:30:56.530440 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:30:56.530628 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:30:56.533845 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:30:56.533864 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:30:56.533882 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:30:56.533937 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:30:56.537214 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:56.538128 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:56.538136 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:56.538139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:56.538296 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:30:56.538692 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:30:56.538696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:30:56.538698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:30:56.539157 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:30:56.539452 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 11:30:56.539820 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 11:30:56.540088 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 11:30:56.541205 8251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:56.633387 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 8128
I20260929 11:30:56.648346 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 7775
I20260929 11:30:56.649552 7838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:30:56.768546 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 7775
I20260929 11:31:16.788811 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:37051
--webserver_interface=127.23.153.62
--webserver_port=41413
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:37051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:31:17.156971 8304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:17.157297 8304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:17.157415 8304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:17.166213 8304 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 11:31:17.166353 8304 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 11:31:17.166420 8304 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 11:31:17.166495 8304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:31:17.166563 8304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:17.166664 8304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:17.166728 8304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:31:17.166810 8304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:31:17.179273 8304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:37051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:37051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=41413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:17.181442 8304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:17.183522 8304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:17.196580 8312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:17.196533 8310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:17.196596 8309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:17.197849 8304 server_base.cc:1034] running on GCE node
I20260929 11:31:17.198946 8304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:17.201195 8304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:17.202477 8304 hybrid_clock.cc:648] HybridClock initialized: now 1790681477202399 us; error 70 us; skew 500 ppm
I20260929 11:31:17.202862 8304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:31:17 dist-test-slave-qk05 krb5kdc[7222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681477, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:31:17.206647 8304 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:31:17.208503 8304 webserver.cc:460] Webserver started at http://127.23.153.62:41413/ using document root <none> and password file <none>
I20260929 11:31:17.209323 8304 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:17.209451 8304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:17.217284 8304 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.004s
I20260929 11:31:17.220835 8319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:17.222760 8304 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260929 11:31:17.223002 8304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "e31d2ca90b2c4b07a097a1b02f45e19a"
format_stamp: "Formatted at 2026-09-29 11:30:31 on dist-test-slave-qk05"
server_key: "f47b31f89441aa9350e8be5efeaf7db3"
server_key_iv: "a4447f83ee612b943d76eaf18913544e"
server_key_version: "encryptionkey@0"
I20260929 11:31:17.224282 8304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:17.252672 8304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:17.258622 8304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:17.258965 8304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:17.279683 8304 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:37051
I20260929 11:31:17.279739 8370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:37051 every 8 connection(s)
I20260929 11:31:17.281394 8304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:31:17.285768 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 8304
I20260929 11:31:17.286705 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:32797
--local_ip_for_outbound_sockets=127.23.153.1
--tserver_master_addrs=127.23.153.62:37051
--webserver_port=44369
--webserver_interface=127.23.153.1
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:31:17.289458 8371 sys_catalog.cc:263] Verifying existing consensus state
I20260929 11:31:17.294845 8371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: Bootstrap starting.
I20260929 11:31:17.320716 8371 log.cc:824] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:17.341609 8371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 11:31:17.342231 8371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: Bootstrap complete.
I20260929 11:31:17.352289 8371 raft_consensus.cc:348] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37051 } }
I20260929 11:31:17.353281 8371 raft_consensus.cc:729] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31d2ca90b2c4b07a097a1b02f45e19a, State: Initialized, Role: FOLLOWER
I20260929 11:31:17.354151 8371 consensus_queue.cc:260] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37051 } }
I20260929 11:31:17.354704 8371 raft_consensus.cc:388] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:31:17.354931 8371 raft_consensus.cc:482] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:31:17.355172 8371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 2 FOLLOWER]: Advancing to term 3
I20260929 11:31:17.360956 8371 raft_consensus.cc:504] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37051 } }
I20260929 11:31:17.361629 8371 leader_election.cc:302] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e31d2ca90b2c4b07a097a1b02f45e19a; no voters:
I20260929 11:31:17.362541 8371 leader_election.cc:288] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 11:31:17.362696 8375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 3 FOLLOWER]: Leader election won for term 3
I20260929 11:31:17.364558 8375 raft_consensus.cc:686] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [term 3 LEADER]: Becoming Leader. State: Replica: e31d2ca90b2c4b07a097a1b02f45e19a, State: Running, Role: LEADER
I20260929 11:31:17.365413 8375 consensus_queue.cc:237] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37051 } }
I20260929 11:31:17.365736 8371 sys_catalog.cc:564] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:31:17.368443 8377 sys_catalog.cc:455] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e31d2ca90b2c4b07a097a1b02f45e19a. Latest consensus state: current_term: 3 leader_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37051 } } }
I20260929 11:31:17.368927 8377 sys_catalog.cc:458] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [sys.catalog]: This master's current role is: LEADER
I20260929 11:31:17.369668 8376 sys_catalog.cc:455] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31d2ca90b2c4b07a097a1b02f45e19a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 37051 } } }
I20260929 11:31:17.370056 8376 sys_catalog.cc:458] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a [sys.catalog]: This master's current role is: LEADER
I20260929 11:31:17.380554 8383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:31:17.392514 8383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:31:17.394248 8383 catalog_manager.cc:1259] Loaded cluster ID: cf153c1eb5ef45c5a0650875b380c8fa
I20260929 11:31:17.394462 8383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:31:17.400055 8383 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:31:17.402791 8383 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: Loaded TSK: 0
I20260929 11:31:17.403343 8383 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e31d2ca90b2c4b07a097a1b02f45e19a: Loaded TSK: 1
F20260929 11:31:17.405195 8383 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f72988229ad google::LogMessage::Fail() at ??:0
@ 0x7f729882588a google::LogMessage::SendToLog() at ??:0
@ 0x7f72988224de google::LogMessage::Flush() at ??:0
@ 0x7f7298823a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f729c5867d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f729c57573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f72ae2511c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f72ae250783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f72ae2dc2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f72ae2dc0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f72ae3265e9 std::function<>::operator()() at ??:0
@ 0x7f72ae254cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f72ae252c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f72ae2da610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f72ae2da439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f729b46e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f729b47191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f729b471799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f729b44ceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f729e7266db start_thread at ??:0
@ 0x7f72976bc61f clone at ??:0
W20260929 11:31:17.742347 8373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:17.742677 8373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:17.742806 8373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:17.751587 8373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:17.751748 8373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:17.751909 8373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:31:17.763294 8373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:32797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=44369
--tserver_master_addrs=127.23.153.62:37051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:17.764739 8373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:17.766253 8373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:17.780736 8418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:17.780732 8421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:17.780753 8419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:17.782076 8373 server_base.cc:1034] running on GCE node
I20260929 11:31:17.783267 8373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:17.784890 8373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:17.786185 8373 hybrid_clock.cc:648] HybridClock initialized: now 1790681477786121 us; error 63 us; skew 500 ppm
I20260929 11:31:17.786584 8373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:31:17 dist-test-slave-qk05 krb5kdc[7222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681477, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:31:17.791150 8373 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:31:17.793272 8373 webserver.cc:460] Webserver started at http://127.23.153.1:44369/ using document root <none> and password file <none>
I20260929 11:31:17.794178 8373 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:17.794339 8373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:17.800735 8373 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 11:31:17.804277 8428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:17.806066 8373 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 11:31:17.806270 8373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "6caa3bf148c743d58cb9fd61f2b75788"
format_stamp: "Formatted at 2026-09-29 11:30:31 on dist-test-slave-qk05"
server_key: "c7e13b43b7d610670214ef2930a0274e"
server_key_iv: "1d4f7c01a03ccfeb0c66543d7f4f0a60"
server_key_version: "encryptionkey@0"
I20260929 11:31:17.807485 8373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:17.842775 8373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:17.848752 8373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:17.849395 8373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:17.850911 8373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:17.852563 8373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:17.852669 8373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:17.852770 8373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:17.852952 8373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:17.889945 8373 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:32797
I20260929 11:31:17.890065 8540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:32797 every 8 connection(s)
I20260929 11:31:17.891577 8373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:31:17.897055 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 8373
I20260929 11:31:17.897598 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:17.895295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:38109)
0929 11:31:17.896177 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:17.896455 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":356,"threads_started":1}
I20260929 11:31:17.897740 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:34211
--local_ip_for_outbound_sockets=127.23.153.2
--tserver_master_addrs=127.23.153.62:37051
--webserver_port=33493
--webserver_interface=127.23.153.2
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:31:17.898418 8541 heartbeater.cc:644] Failed to heartbeat to 127.23.153.62:37051 (0 consecutive failures): Network error: Failed to ping master at 127.23.153.62:37051: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
I20260929 11:31:17.899451 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:17.899075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:57497)
0929 11:31:17.899247 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:17.899326 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 11:31:17.900440 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:17.900039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:57103)
0929 11:31:17.900210 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:17.900307 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 11:31:17.900707 8541 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 11:31:18.268704 8544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:18.269049 8544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:18.269172 8544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:18.277751 8544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:18.277909 8544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:18.278080 8544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:31:18.292059 8544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:34211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=33493
--tserver_master_addrs=127.23.153.62:37051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:18.294164 8544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:18.295828 8544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:18.310786 8550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:18.310854 8549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:18.310811 8552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:18.312022 8544 server_base.cc:1034] running on GCE node
I20260929 11:31:18.312718 8544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:18.313882 8544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:18.315165 8544 hybrid_clock.cc:648] HybridClock initialized: now 1790681478315108 us; error 45 us; skew 500 ppm
I20260929 11:31:18.315550 8544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:31:18 dist-test-slave-qk05 krb5kdc[7222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681478, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:31:18.319205 8544 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:31:18.320900 8544 webserver.cc:460] Webserver started at http://127.23.153.2:33493/ using document root <none> and password file <none>
I20260929 11:31:18.321628 8544 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:18.321753 8544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:18.328235 8544 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:31:18.331314 8559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:18.333298 8544 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 11:31:18.333498 8544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "1647fd55afa645ce933df9f1fbe15d9b"
format_stamp: "Formatted at 2026-09-29 11:30:32 on dist-test-slave-qk05"
server_key: "a36f4b8cdbbaa37312648ae636558107"
server_key_iv: "97acf3669cc0ce421a56f6f720acb3d6"
server_key_version: "encryptionkey@0"
I20260929 11:31:18.334611 8544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:18.351263 8544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:18.356036 8544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:18.356333 8544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:18.357429 8544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:18.359068 8544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:18.359185 8544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 11:31:18.359308 8544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:18.359378 8544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:18.393411 8544 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:34211
I20260929 11:31:18.393422 8671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:34211 every 8 connection(s)
I20260929 11:31:18.395095 8544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:31:18.400769 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:18.398444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:42903)
0929 11:31:18.399363 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:18.399622 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":368,"threads_started":1}
W20260929 11:31:18.401537 8672 heartbeater.cc:644] Failed to heartbeat to 127.23.153.62:37051 (0 consecutive failures): Network error: Failed to ping master at 127.23.153.62:37051: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
I20260929 11:31:18.402482 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:18.402117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:49455)
0929 11:31:18.402285 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:18.402360 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 11:31:18.403378 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:18.403043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:36651)
0929 11:31:18.403175 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:18.403262 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 11:31:18.403604 8672 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 11:31:18.404676 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 8544
I20260929 11:31:18.405288 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:42541
--local_ip_for_outbound_sockets=127.23.153.3
--tserver_master_addrs=127.23.153.62:37051
--webserver_port=37113
--webserver_interface=127.23.153.3
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:31:18.762584 8675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:18.762921 8675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:18.763067 8675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:18.771727 8675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:18.771886 8675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:18.772051 8675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:31:18.782966 8675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:42541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=37113
--tserver_master_addrs=127.23.153.62:37051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:18.784366 8675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:18.785727 8675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:18.800628 8680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:18.800580 8683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:18.800551 8681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:18.802055 8675 server_base.cc:1034] running on GCE node
I20260929 11:31:18.802753 8675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:18.804131 8675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:18.805408 8675 hybrid_clock.cc:648] HybridClock initialized: now 1790681478805338 us; error 56 us; skew 500 ppm
I20260929 11:31:18.805778 8675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:31:18 dist-test-slave-qk05 krb5kdc[7222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681478, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:31:18.809556 8675 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:31:18.811324 8675 webserver.cc:460] Webserver started at http://127.23.153.3:37113/ using document root <none> and password file <none>
I20260929 11:31:18.812096 8675 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:18.812240 8675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:18.818737 8675 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 11:31:18.822069 8690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:18.823844 8675 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:31:18.824052 8675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "ee8ebbd928654b818d22b6bef42dd74c"
format_stamp: "Formatted at 2026-09-29 11:30:32 on dist-test-slave-qk05"
server_key: "d1cfe413b86861c5b7109e5468d4efa1"
server_key_iv: "7972e6d9b790071eb4db6a844c032679"
server_key_version: "encryptionkey@0"
I20260929 11:31:18.825124 8675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:18.841495 8675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:18.846050 8675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:18.846340 8675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:18.847493 8675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:18.849044 8675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:18.849152 8675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:18.849272 8675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:18.849355 8675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:18.883620 8675 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:42541
I20260929 11:31:18.883693 8802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:42541 every 8 connection(s)
I20260929 11:31:18.885157 8675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:31:18.890741 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 8675
I20260929 11:31:18.890910 8805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:18.888430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:39175)
0929 11:31:18.889314 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:18.889625 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":318,"threads_started":1}
W20260929 11:31:18.891880 8803 heartbeater.cc:644] Failed to heartbeat to 127.23.153.62:37051 (0 consecutive failures): Network error: Failed to ping master at 127.23.153.62:37051: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
I20260929 11:31:18.893052 8805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:18.892496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:46431)
0929 11:31:18.892677 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:18.892768 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 11:31:18.894572 8805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:18.894104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:57057)
0929 11:31:18.894269 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:18.894437 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 11:31:18.894869 8803 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 11:31:18.902269 8807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:18.901425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:49363)
0929 11:31:18.901928 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:18.902029 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":244,"threads_started":1}
I20260929 11:31:19.405150 8808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:19.404276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:43687)
0929 11:31:19.404768 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:19.404913 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":233,"threads_started":1}
I20260929 11:31:19.896412 8809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:19.895506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:39795)
0929 11:31:19.896042 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:19.896166 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":243,"threads_started":1}
I20260929 11:31:19.904385 8810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:19.903464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:33705)
0929 11:31:19.904098 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:19.904197 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":279,"threads_started":1}
I20260929 11:31:20.406900 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:20.406135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:46065)
0929 11:31:20.406621 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:20.406711 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":237,"threads_started":1}
I20260929 11:31:20.898193 8812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:20.897468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:52331)
0929 11:31:20.897928 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:20.898019 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":219,"threads_started":1}
I20260929 11:31:20.906091 8813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:20.905326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:45517)
0929 11:31:20.905803 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:20.905905 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":251,"threads_started":1}
I20260929 11:31:21.408742 8814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:21.407982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:35585)
0929 11:31:21.408487 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:21.408570 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":270,"threads_started":1}
I20260929 11:31:21.899911 8815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:21.899170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:43725)
0929 11:31:21.899648 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:21.899732 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":256,"threads_started":1}
I20260929 11:31:21.907981 8816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:21.907166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:36555)
0929 11:31:21.907707 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:21.907800 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":283,"threads_started":1}
I20260929 11:31:22.410563 8817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:22.409743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:56397)
0929 11:31:22.410267 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:22.410363 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":268,"threads_started":1}
I20260929 11:31:22.901928 8818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:22.901041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:45509)
0929 11:31:22.901528 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:22.901621 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":246,"threads_started":1}
I20260929 11:31:22.909871 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:22.909023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:57785)
0929 11:31:22.909569 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:22.909666 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":265,"threads_started":1}
I20260929 11:31:23.412984 8820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:23.411696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:60155)
0929 11:31:23.412559 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:23.412703 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":498,"threads_started":1}
I20260929 11:31:23.903918 8821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:23.903078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:60211)
0929 11:31:23.903623 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:23.903724 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":274,"threads_started":1}
I20260929 11:31:23.911737 8822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:23.910909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:55821)
0929 11:31:23.911450 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:23.911565 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":251,"threads_started":1}
I20260929 11:31:24.415578 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:24.414394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:57801)
0929 11:31:24.415256 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:24.415387 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":519,"threads_started":1}
I20260929 11:31:24.905743 8824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:24.904968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:60659)
0929 11:31:24.905472 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:24.905572 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":272,"threads_started":1}
I20260929 11:31:24.914315 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:24.912804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:53133)
0929 11:31:24.913895 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:24.914035 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":916,"thread_start_us":650,"threads_started":1}
I20260929 11:31:25.417820 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:25.416892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:38253)
0929 11:31:25.417438 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:25.417532 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":256,"threads_started":1}
I20260929 11:31:25.908203 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:25.907264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:45365)
0929 11:31:25.907819 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:25.907920 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":223,"threads_started":1}
I20260929 11:31:25.916148 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:25.915369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:57103)
0929 11:31:25.915869 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:25.915969 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":241,"threads_started":1}
I20260929 11:31:26.419703 8829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:26.418895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:55639)
0929 11:31:26.419435 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:26.419530 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":288,"threads_started":1}
I20260929 11:31:26.909950 8830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:26.909194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:33289)
0929 11:31:26.909664 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:26.909774 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":240,"threads_started":1}
I20260929 11:31:26.918087 8831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:26.917223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:55193)
0929 11:31:26.917782 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:26.917910 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":284,"threads_started":1}
I20260929 11:31:27.421679 8832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:27.420867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:50203)
0929 11:31:27.421399 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:27.421497 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":264,"threads_started":1}
I20260929 11:31:27.911868 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:27.910942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:47439)
0929 11:31:27.911487 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:27.911610 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":271,"threads_started":1}
I20260929 11:31:27.920255 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:27.919408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:51543)
0929 11:31:27.919908 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:27.920045 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":274,"threads_started":1}
I20260929 11:31:28.424391 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:28.423215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:60931)
0929 11:31:28.424044 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:28.424186 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":369,"threads_started":1}
I20260929 11:31:28.913846 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:28.912995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:53595)
0929 11:31:28.913477 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:28.913599 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":256,"threads_started":1}
I20260929 11:31:28.922070 8837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:28.921273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:48061)
0929 11:31:28.921744 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:28.921879 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":246,"threads_started":1}
I20260929 11:31:29.426429 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:29.425580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:59943)
0929 11:31:29.426109 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:29.426214 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":266,"threads_started":1}
I20260929 11:31:29.915709 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:29.914937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:45247)
0929 11:31:29.915424 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:29.915523 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":261,"threads_started":1}
I20260929 11:31:29.923926 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:29.923177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:36383)
0929 11:31:29.923667 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:29.923765 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":221,"threads_started":1}
I20260929 11:31:30.428354 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:30.427538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:51819)
0929 11:31:30.428086 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:30.428185 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":292,"threads_started":1}
I20260929 11:31:30.917596 8842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:30.916885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:45243)
0929 11:31:30.917352 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:30.917436 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":226,"threads_started":1}
I20260929 11:31:30.925945 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:30.925042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:35853)
0929 11:31:30.925638 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:30.925748 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":328,"threads_started":1}
I20260929 11:31:31.430292 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:31.429462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:48255)
0929 11:31:31.429998 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:31.430107 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":273,"threads_started":1}
I20260929 11:31:31.919709 8845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:31.918836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:58865)
0929 11:31:31.919406 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:31.919516 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":307,"threads_started":1}
I20260929 11:31:31.927793 8846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:31.926979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:41691)
0929 11:31:31.927481 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:31.927577 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":225,"threads_started":1}
I20260929 11:31:32.432137 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:32.431346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:59343)
0929 11:31:32.431844 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:32.431941 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":232,"threads_started":1}
I20260929 11:31:32.921478 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:32.920706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:58411)
0929 11:31:32.921181 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:32.921286 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":237,"threads_started":1}
I20260929 11:31:32.929917 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:32.929111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.1:41769)
0929 11:31:32.929574 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:32.929677 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":233,"threads_started":1}
I20260929 11:31:33.434309 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:33.433442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.2:49181)
0929 11:31:33.434010 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:33.434112 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":353,"threads_started":1}
I20260929 11:31:33.895154 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 8373
I20260929 11:31:33.918560 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:33.896400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37982 (local address 127.23.153.1:32797)
0929 11:31:33.897511 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:33.897526 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:33.897565 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:31:33.899233 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:33.899244 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:33.899755 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:33.900082 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:33.901690 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:33.902973 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:33.904368 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:33.905112 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:33.905262 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:33.911831 (+ 6569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:33.911843 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:33.911863 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:33.911933 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:33.915160 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:33.916047 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:33.916055 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:33.916063 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:33.916283 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:33.916607 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:33.916610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:33.916612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:33.917195 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:33.917460 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:33.917807 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 11:31:33.918267 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":962,"thread_start_us":772,"threads_started":1}
I20260929 11:31:33.920073 8536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:33.923730 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:33.922639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:37051 (local address 127.23.153.3:57465)
0929 11:31:33.923276 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:33.923417 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.62:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":263,"threads_started":1}
I20260929 11:31:34.008625 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 8373
I20260929 11:31:34.024091 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 8544
I20260929 11:31:34.044783 8855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:34.025052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42650 (local address 127.23.153.2:34211)
0929 11:31:34.025551 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:34.025567 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:34.025608 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:31:34.027244 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:34.027256 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:34.027742 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:34.028076 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:34.029736 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:34.031335 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:34.033203 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:34.034235 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:34.034461 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:34.037700 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:34.037711 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:34.037731 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:34.037789 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:34.041084 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:34.041995 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:34.042003 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:34.042011 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:34.042233 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:34.042554 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:34.042556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:34.042558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:34.043228 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:34.043486 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:34.043992 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 11:31:34.044464 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":199,"threads_started":1}
I20260929 11:31:34.046326 8667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:34.133376 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 8544
I20260929 11:31:34.151846 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 8675
I20260929 11:31:34.174130 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:34.152881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41570 (local address 127.23.153.3:42541)
0929 11:31:34.153508 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:34.153523 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:34.153564 (+ 41us) server_negotiation.cc:407] Connection header received
0929 11:31:34.155309 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:34.155323 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:34.155814 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:34.156180 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:34.157717 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:34.158987 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:34.160404 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:34.161196 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:34.161391 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:34.164685 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:34.164704 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:34.164725 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:34.164788 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:34.168773 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:34.169671 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:34.169680 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:34.169688 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:34.169919 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:34.170339 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:34.170343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:34.170345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:34.170986 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:34.171874 (+ 888us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:34.172424 (+ 550us) server_negotiation.cc:299] Negotiation successful
0929 11:31:34.173250 (+ 826us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":300,"threads_started":1}
I20260929 11:31:34.175987 8798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:34.263713 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 8675
2026-09-29T11:31:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64892 ms)
[----------] 30 tests from SecurityITest (313066 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:31:34 dist-test-slave-qk05 krb5kdc[8861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:31:34 dist-test-slave-qk05 krb5kdc[8861](info): set up 2 sockets
Sep 29 11:31:34 dist-test-slave-qk05 krb5kdc[8861](info): commencing operation
krb5kdc: starting...
W20260929 11:31:35.324014 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:31:35 dist-test-slave-qk05 krb5kdc[8861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:31:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:31:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:31:35.584887 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:42487
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:38005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:42487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:31:35.953275 8874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:35.953615 8874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:35.953720 8874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:35.961794 8874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:31:35.961937 8874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:35.961987 8874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:35.962044 8874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:31:35.962087 8874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:31:35.972723 8874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:38005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:42487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:42487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:35.974110 8874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:35.975497 8874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:35.988731 8879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:35.988688 8880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:35.988982 8882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:35.990085 8874 server_base.cc:1034] running on GCE node
I20260929 11:31:35.990965 8874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:35.992648 8874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:35.993897 8874 hybrid_clock.cc:648] HybridClock initialized: now 1790681495993841 us; error 42 us; skew 500 ppm
I20260929 11:31:35.994302 8874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:31:35 dist-test-slave-qk05 krb5kdc[8861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681495, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:31:35.997953 8874 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:31:35.999840 8874 webserver.cc:460] Webserver started at http://127.23.153.62:46333/ using document root <none> and password file <none>
I20260929 11:31:36.000676 8874 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:36.000840 8874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:36.001173 8874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:36.004603 8874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "95eaf5f4fa46493e81f5f92ae0f9b050"
format_stamp: "Formatted at 2026-09-29 11:31:35 on dist-test-slave-qk05"
server_key: "6e4361c51dc14f98f11c6f638b2adc23"
server_key_iv: "3dc912ee5b8730aa71fb93e2628d1773"
server_key_version: "encryptionkey@0"
I20260929 11:31:36.005668 8874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "95eaf5f4fa46493e81f5f92ae0f9b050"
format_stamp: "Formatted at 2026-09-29 11:31:35 on dist-test-slave-qk05"
server_key: "6e4361c51dc14f98f11c6f638b2adc23"
server_key_iv: "3dc912ee5b8730aa71fb93e2628d1773"
server_key_version: "encryptionkey@0"
I20260929 11:31:36.012307 8874 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 11:31:36.016660 8889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:36.018354 8874 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:31:36.018580 8874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "95eaf5f4fa46493e81f5f92ae0f9b050"
format_stamp: "Formatted at 2026-09-29 11:31:35 on dist-test-slave-qk05"
server_key: "6e4361c51dc14f98f11c6f638b2adc23"
server_key_iv: "3dc912ee5b8730aa71fb93e2628d1773"
server_key_version: "encryptionkey@0"
I20260929 11:31:36.018838 8874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:36.048331 8874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:36.053700 8874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:36.054023 8874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:36.074887 8874 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:42487
I20260929 11:31:36.074893 8940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:42487 every 8 connection(s)
I20260929 11:31:36.076427 8874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:31:36.080590 8941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:36.082193 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 8874
I20260929 11:31:36.082405 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:31:36.083099 24164 external_mini_cluster.cc:1428] Setting key 44694bef37eb65b2db364549a100f609
I20260929 11:31:36.094375 8941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050: Bootstrap starting.
I20260929 11:31:36.098970 8941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050: Neither blocks nor log segments found. Creating new log.
Sep 29 11:31:36 dist-test-slave-qk05 krb5kdc[8861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:31:36.100683 8941 log.cc:824] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:36.104401 8941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050: No bootstrap required, opened a new log
I20260929 11:31:36.112648 8944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:36.085247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39598 (local address 127.23.153.62:42487)
0929 11:31:36.086158 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:36.086185 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:36.086249 (+ 64us) server_negotiation.cc:407] Connection header received
0929 11:31:36.088369 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:36.088401 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:36.089045 (+ 644us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:36.089452 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:36.091506 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:36.092978 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:36.094954 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:36.095817 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:36.095983 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:36.101900 (+ 5917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:36.101915 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:36.101940 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:36.102004 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:36.105823 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:36.107656 (+ 1833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:36.107673 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:36.107688 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:36.108041 (+ 353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:36.108823 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:36.108830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:36.108833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:36.109594 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:36.109881 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:36.110507 (+ 626us) server_negotiation.cc:299] Negotiation successful
0929 11:31:36.111385 (+ 878us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":425,"threads_started":1}
I20260929 11:31:36.115720 8941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95eaf5f4fa46493e81f5f92ae0f9b050" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42487 } }
I20260929 11:31:36.116241 8941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:36.116463 8941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95eaf5f4fa46493e81f5f92ae0f9b050, State: Initialized, Role: FOLLOWER
I20260929 11:31:36.117161 8941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95eaf5f4fa46493e81f5f92ae0f9b050" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42487 } }
I20260929 11:31:36.117408 8941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:31:36.117511 8941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:31:36.117650 8941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:36.120142 8941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95eaf5f4fa46493e81f5f92ae0f9b050" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42487 } }
I20260929 11:31:36.120697 8941 leader_election.cc:302] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95eaf5f4fa46493e81f5f92ae0f9b050; no voters:
I20260929 11:31:36.121330 8941 leader_election.cc:288] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:31:36.121484 8946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:31:36.122553 8946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050 [term 1 LEADER]: Becoming Leader. State: Replica: 95eaf5f4fa46493e81f5f92ae0f9b050, State: Running, Role: LEADER
I20260929 11:31:36.123306 8946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95eaf5f4fa46493e81f5f92ae0f9b050" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42487 } }
I20260929 11:31:36.123811 8941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:31:36.126796 8947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95eaf5f4fa46493e81f5f92ae0f9b050" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95eaf5f4fa46493e81f5f92ae0f9b050" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42487 } } }
I20260929 11:31:36.127326 8947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050 [sys.catalog]: This master's current role is: LEADER
I20260929 11:31:36.129158 8948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95eaf5f4fa46493e81f5f92ae0f9b050. Latest consensus state: current_term: 1 leader_uuid: "95eaf5f4fa46493e81f5f92ae0f9b050" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95eaf5f4fa46493e81f5f92ae0f9b050" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42487 } } }
I20260929 11:31:36.129628 8948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050 [sys.catalog]: This master's current role is: LEADER
I20260929 11:31:36.130142 8954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:31:36.137012 8954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:31:36.148463 8954 catalog_manager.cc:1347] Generated new cluster ID: 0714c8878df64fed82fba32a54c14c35
I20260929 11:31:36.148622 8954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:31:36.158409 8954 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:31:36.159984 8954 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:31:36.171414 8954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050: Generated new TSK 0
I20260929 11:31:36.172688 8954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
I20260929 11:31:36.245141 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42487
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:38005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:31:36.610651 8967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:36.611006 8967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:36.611172 8967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:36.619379 8967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:36.619539 8967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:36.619697 8967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:31:36.630643 8967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:38005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:36.632195 8967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:36.633677 8967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:36.648475 8972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:36.648430 8975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:36.648772 8973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:36.649459 8967 server_base.cc:1034] running on GCE node
I20260929 11:31:36.650138 8967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:36.651242 8967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:36.652474 8967 hybrid_clock.cc:648] HybridClock initialized: now 1790681496652428 us; error 42 us; skew 500 ppm
I20260929 11:31:36.652829 8967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:31:36 dist-test-slave-qk05 krb5kdc[8861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681496, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:31:36.657207 8967 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:31:36.659004 8967 webserver.cc:460] Webserver started at http://127.23.153.1:34471/ using document root <none> and password file <none>
I20260929 11:31:36.659821 8967 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:36.659960 8967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:36.660291 8967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:36.663635 8967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "1f1daa4acf1147fc9c632be4bed43673"
format_stamp: "Formatted at 2026-09-29 11:31:36 on dist-test-slave-qk05"
server_key: "cfd841bbd4539233bccc1ce44a20e275"
server_key_iv: "4e144129306bac565f420d0f73fbd49d"
server_key_version: "encryptionkey@0"
I20260929 11:31:36.664602 8967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "1f1daa4acf1147fc9c632be4bed43673"
format_stamp: "Formatted at 2026-09-29 11:31:36 on dist-test-slave-qk05"
server_key: "cfd841bbd4539233bccc1ce44a20e275"
server_key_iv: "4e144129306bac565f420d0f73fbd49d"
server_key_version: "encryptionkey@0"
I20260929 11:31:36.672953 8967 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260929 11:31:36.677752 8982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:36.679565 8967 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:31:36.679749 8967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "1f1daa4acf1147fc9c632be4bed43673"
format_stamp: "Formatted at 2026-09-29 11:31:36 on dist-test-slave-qk05"
server_key: "cfd841bbd4539233bccc1ce44a20e275"
server_key_iv: "4e144129306bac565f420d0f73fbd49d"
server_key_version: "encryptionkey@0"
I20260929 11:31:36.679962 8967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:36.696501 8967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:36.700835 8967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:36.701133 8967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:36.702196 8967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:36.703698 8967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:36.703775 8967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:36.703876 8967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:36.703919 8967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:36.736342 8967 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:39445
I20260929 11:31:36.736382 9094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:39445 every 8 connection(s)
I20260929 11:31:36.737692 8967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:31:36.744223 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 8967
I20260929 11:31:36.744454 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:31:36.745112 24164 external_mini_cluster.cc:1428] Setting key e5f26b91fe79b81996e636ce600ac85f
Sep 29 11:31:36 dist-test-slave-qk05 krb5kdc[8861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681496, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:31:36.760614 9097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:36.740508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:36845 (local address 127.23.153.62:42487)
0929 11:31:36.740988 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:36.740993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:36.742056 (+ 1063us) server_negotiation.cc:407] Connection header received
0929 11:31:36.744160 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:36.744167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:36.744277 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:36.744414 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:36.746449 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:36.747434 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:36.749207 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:36.749785 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:36.749921 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:36.753676 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:36.753685 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:36.753690 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:36.753736 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:36.756966 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:36.758052 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:36.758056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:36.758058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:36.758210 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:36.758726 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:36.758729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:36.758731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:36.759008 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:36.759195 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:36.760120 (+ 925us) server_negotiation.cc:299] Negotiation successful
0929 11:31:36.760338 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":240,"threads_started":1}
I20260929 11:31:36.762094 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:36.741068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42487 (local address 127.23.153.1:36845)
0929 11:31:36.741833 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:36.741897 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:36.743814 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:36.744737 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:36.744747 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:36.745385 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:31:36.746272 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:36.746298 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:36.747621 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:36.747626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:36.749020 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:36.749037 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:36.750027 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:36.750032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:36.750262 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:36.750958 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:31:36.750983 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:31:36.753357 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:31:36.757194 (+ 3837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:31:36.757211 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:31:36.757233 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:31:36.757915 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:31:36.758320 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:31:36.758326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:31:36.758329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:31:36.758613 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:31:36.759164 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:31:36.759173 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:31:36.759678 (+ 505us) client_negotiation.cc:769] Sending connection context
0929 11:31:36.760011 (+ 333us) client_negotiation.cc:240] Negotiation successful
0929 11:31:36.760604 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":269,"threads_started":1}
I20260929 11:31:36.764040 9095 heartbeater.cc:347] Connected to a master server at 127.23.153.62:42487
I20260929 11:31:36.764400 9095 heartbeater.cc:464] Registering TS with master...
I20260929 11:31:36.765252 9095 heartbeater.cc:511] Master 127.23.153.62:42487 requested a full tablet report, sending...
I20260929 11:31:36.767867 8906 ts_manager.cc:194] Registered new tserver with Master: 1f1daa4acf1147fc9c632be4bed43673 (127.23.153.1:39445)
I20260929 11:31:36.769464 8906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:36845
WARNING: no policy specified for kudu/127.23.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
I20260929 11:31:36.804740 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42487
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:38005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:31:37.170536 9101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:37.170869 9101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:37.170969 9101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:37.178628 9101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:37.178768 9101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:37.178890 9101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:31:37.189344 9101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:38005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:37.190698 9101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:37.192214 9101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:37.207260 9106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:37.207291 9107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:37.207321 9109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:37.207708 9101 server_base.cc:1034] running on GCE node
I20260929 11:31:37.208535 9101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:37.209542 9101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:37.210753 9101 hybrid_clock.cc:648] HybridClock initialized: now 1790681497210699 us; error 42 us; skew 500 ppm
I20260929 11:31:37.211164 9101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:31:37 dist-test-slave-qk05 krb5kdc[8861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681497, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:31:37.214623 9101 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:31:37.216586 9101 webserver.cc:460] Webserver started at http://127.23.153.2:38613/ using document root <none> and password file <none>
I20260929 11:31:37.217356 9101 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:37.217480 9101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:37.217789 9101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:37.221076 9101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "a2e163380cda4c23a35152aa5312346e"
format_stamp: "Formatted at 2026-09-29 11:31:37 on dist-test-slave-qk05"
server_key: "2550df59df3a58c4429170c5bf5f05f9"
server_key_iv: "141fa2d3d1fd017d9ad895f2030ef587"
server_key_version: "encryptionkey@0"
I20260929 11:31:37.221925 9101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "a2e163380cda4c23a35152aa5312346e"
format_stamp: "Formatted at 2026-09-29 11:31:37 on dist-test-slave-qk05"
server_key: "2550df59df3a58c4429170c5bf5f05f9"
server_key_iv: "141fa2d3d1fd017d9ad895f2030ef587"
server_key_version: "encryptionkey@0"
I20260929 11:31:37.228155 9101 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 11:31:37.232335 9116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:37.234041 9101 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 11:31:37.234256 9101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "a2e163380cda4c23a35152aa5312346e"
format_stamp: "Formatted at 2026-09-29 11:31:37 on dist-test-slave-qk05"
server_key: "2550df59df3a58c4429170c5bf5f05f9"
server_key_iv: "141fa2d3d1fd017d9ad895f2030ef587"
server_key_version: "encryptionkey@0"
I20260929 11:31:37.234489 9101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:37.257121 9101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:37.262168 9101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:37.262486 9101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:37.263681 9101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:37.265374 9101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:37.265463 9101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:37.265588 9101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:37.265651 9101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:37.302317 9101 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:39863
I20260929 11:31:37.302368 9228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:39863 every 8 connection(s)
I20260929 11:31:37.303798 9101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:31:37.312022 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 9101
I20260929 11:31:37.312417 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:31:37.313079 24164 external_mini_cluster.cc:1428] Setting key 0f7af573f51072ee68bb5aef95752fd3
Sep 29 11:31:37 dist-test-slave-qk05 krb5kdc[8861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681497, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:31:37.329177 9231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:37.307178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:40403 (local address 127.23.153.62:42487)
0929 11:31:37.307673 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:37.307678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:37.308725 (+ 1047us) server_negotiation.cc:407] Connection header received
0929 11:31:37.310883 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:37.310891 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:37.310980 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:37.311130 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:37.313001 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.313841 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.316297 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.317084 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.317244 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:37.322051 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:37.322059 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:37.322062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:37.322111 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:37.325436 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:37.326423 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:37.326429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:37.326433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:37.326636 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:37.327230 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:37.327235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:37.327238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:37.327613 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:37.327764 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:37.328545 (+ 781us) server_negotiation.cc:299] Negotiation successful
0929 11:31:37.328800 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":258,"threads_started":1}
I20260929 11:31:37.330070 9232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:37.307682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42487 (local address 127.23.153.2:40403)
0929 11:31:37.308500 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:37.308564 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:37.310503 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:37.311424 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:37.311434 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:37.312071 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:31:37.312784 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:37.312808 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.314035 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.314039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:37.316079 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:37.316099 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.317364 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.317370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:37.317607 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:37.318372 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:31:37.318404 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:31:37.321541 (+ 3137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:31:37.325662 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:31:37.325675 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:31:37.325698 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:31:37.326261 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:31:37.326803 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:31:37.326808 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:31:37.326810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:31:37.327099 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:31:37.327729 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:31:37.327736 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:31:37.328139 (+ 403us) client_negotiation.cc:769] Sending connection context
0929 11:31:37.328404 (+ 265us) client_negotiation.cc:240] Negotiation successful
0929 11:31:37.328857 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":304,"threads_started":1}
I20260929 11:31:37.331954 9229 heartbeater.cc:347] Connected to a master server at 127.23.153.62:42487
I20260929 11:31:37.332361 9229 heartbeater.cc:464] Registering TS with master...
I20260929 11:31:37.333184 9229 heartbeater.cc:511] Master 127.23.153.62:42487 requested a full tablet report, sending...
I20260929 11:31:37.334968 8906 ts_manager.cc:194] Registered new tserver with Master: a2e163380cda4c23a35152aa5312346e (127.23.153.2:39863)
I20260929 11:31:37.336058 8906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:40403
WARNING: no policy specified for kudu/127.23.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
I20260929 11:31:37.375923 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42487
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:38005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:31:37.735302 9235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:37.735734 9235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:37.735903 9235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:37.743963 9235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:37.744122 9235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:37.744287 9235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:31:37.755632 9235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:38005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:42487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:37.757098 9235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:37.758512 9235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:37.773240 9241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:37.773478 9095 heartbeater.cc:503] Master 127.23.153.62:42487 was elected leader, sending a full tablet report...
W20260929 11:31:37.773326 9243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:37.773923 9240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:37.774374 9235 server_base.cc:1034] running on GCE node
I20260929 11:31:37.775241 9235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:37.776418 9235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:37.777721 9235 hybrid_clock.cc:648] HybridClock initialized: now 1790681497777642 us; error 53 us; skew 500 ppm
I20260929 11:31:37.778095 9235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:31:37 dist-test-slave-qk05 krb5kdc[8861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681497, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:31:37.781810 9235 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:31:37.784324 9235 webserver.cc:460] Webserver started at http://127.23.153.3:37669/ using document root <none> and password file <none>
I20260929 11:31:37.785279 9235 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:37.785423 9235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:37.785840 9235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:37.789287 9235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "600f7340b94842268b54b56d605755d0"
format_stamp: "Formatted at 2026-09-29 11:31:37 on dist-test-slave-qk05"
server_key: "7e9b8e7a0b167095ad3feb59d1d90caf"
server_key_iv: "2ac3fabc2c37359ddf8f2845651ca2ac"
server_key_version: "encryptionkey@0"
I20260929 11:31:37.790179 9235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "600f7340b94842268b54b56d605755d0"
format_stamp: "Formatted at 2026-09-29 11:31:37 on dist-test-slave-qk05"
server_key: "7e9b8e7a0b167095ad3feb59d1d90caf"
server_key_iv: "2ac3fabc2c37359ddf8f2845651ca2ac"
server_key_version: "encryptionkey@0"
I20260929 11:31:37.796736 9235 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 11:31:37.801295 9250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:37.803547 9235 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 11:31:37.803901 9235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "600f7340b94842268b54b56d605755d0"
format_stamp: "Formatted at 2026-09-29 11:31:37 on dist-test-slave-qk05"
server_key: "7e9b8e7a0b167095ad3feb59d1d90caf"
server_key_iv: "2ac3fabc2c37359ddf8f2845651ca2ac"
server_key_version: "encryptionkey@0"
I20260929 11:31:37.804158 9235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:37.823210 9235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:37.828683 9235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:37.828999 9235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:37.830135 9235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:37.831911 9235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:37.831996 9235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:37.832103 9235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:37.832178 9235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:37.871384 9235 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:46817
I20260929 11:31:37.871397 9362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:46817 every 8 connection(s)
I20260929 11:31:37.872807 9235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:31:37.873574 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 9235
I20260929 11:31:37.873817 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:31:37.874370 24164 external_mini_cluster.cc:1428] Setting key 54b1a450213c5abf8715c173fbf32685
Sep 29 11:31:37 dist-test-slave-qk05 krb5kdc[8861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681497, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:31:37.895732 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:37.875682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:50065 (local address 127.23.153.62:42487)
0929 11:31:37.876153 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:37.876159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:37.877639 (+ 1480us) server_negotiation.cc:407] Connection header received
0929 11:31:37.879733 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:37.879738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:37.879815 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:37.879916 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:37.881875 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.882763 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.884729 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.885226 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.885355 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:37.889602 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:37.889608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:37.889613 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:37.889660 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:37.892341 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:37.893230 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:37.893235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:37.893237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:37.893376 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:37.893921 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:37.893925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:37.893927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:37.894240 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:37.894369 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:37.895222 (+ 853us) server_negotiation.cc:299] Negotiation successful
0929 11:31:37.895420 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":220,"threads_started":1}
I20260929 11:31:37.896693 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:37.876439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:42487 (local address 127.23.153.3:50065)
0929 11:31:37.877462 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:37.877513 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:37.879364 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:37.880150 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:37.880159 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:37.880821 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:31:37.881643 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:37.881673 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.882964 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.882970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:37.884540 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:37.884558 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.885485 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.885493 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:37.885866 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:37.886618 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:31:37.886645 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:31:37.889359 (+ 2714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:31:37.892511 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:31:37.892525 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:31:37.892541 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:31:37.893054 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:31:37.893509 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:31:37.893514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:31:37.893517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:31:37.893773 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:31:37.894385 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:31:37.894392 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:31:37.894788 (+ 396us) client_negotiation.cc:769] Sending connection context
0929 11:31:37.895119 (+ 331us) client_negotiation.cc:240] Negotiation successful
0929 11:31:37.895568 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":458,"threads_started":1}
I20260929 11:31:37.898216 9363 heartbeater.cc:347] Connected to a master server at 127.23.153.62:42487
I20260929 11:31:37.898548 9363 heartbeater.cc:464] Registering TS with master...
I20260929 11:31:37.899654 9363 heartbeater.cc:511] Master 127.23.153.62:42487 requested a full tablet report, sending...
I20260929 11:31:37.901408 8906 ts_manager.cc:194] Registered new tserver with Master: 600f7340b94842268b54b56d605755d0 (127.23.153.3:46817)
I20260929 11:31:37.902487 8906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:50065
I20260929 11:31:37.903249 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:31:37.922634 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:37.907984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58738 (local address 127.23.153.62:42487)
0929 11:31:37.908221 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:37.908228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:37.908880 (+ 652us) server_negotiation.cc:407] Connection header received
0929 11:31:37.909175 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:37.909180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:37.909266 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:37.909397 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:37.910973 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.911900 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.913319 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.913908 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.914033 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:37.916652 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:37.916659 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:37.916679 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:37.916728 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:37.919551 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:37.920507 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:37.920513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:37.920515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:37.920677 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:37.921035 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:37.921038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:37.921040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:37.921396 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:37.921606 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:37.922056 (+ 450us) server_negotiation.cc:299] Negotiation successful
0929 11:31:37.922285 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:31:37.927433 8906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:31:37.934885 8906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:31:37.961972 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:37.949538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:39863 (local address 127.0.0.1:55642)
0929 11:31:37.952934 (+ 3396us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:37.952951 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:37.953082 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:37.953658 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:37.953666 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:37.953719 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:37.954195 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:37.954211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.956252 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.956256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:37.958935 (+ 2679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:37.958953 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.960793 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.960800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:37.961049 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:37.961118 (+ 69us) client_negotiation.cc:769] Sending connection context
0929 11:31:37.961259 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 11:31:37.961475 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3272,"mutex_wait_us":23,"spinlock_wait_cycles":5504,"thread_start_us":3058,"threads_started":1}
I20260929 11:31:37.961967 9378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:37.949888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55642 (local address 127.23.153.2:39863)
0929 11:31:37.952449 (+ 2561us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:37.952460 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:37.953059 (+ 599us) server_negotiation.cc:407] Connection header received
0929 11:31:37.953226 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:37.953232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:37.953436 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:37.953616 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:37.954392 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.956073 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.959198 (+ 3125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.960574 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.960714 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:37.960955 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:37.961354 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 11:31:37.961627 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2360,"thread_start_us":2305,"threads_started":1}
I20260929 11:31:37.964618 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:37.949288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:46817 (local address 127.0.0.1:41500)
0929 11:31:37.950277 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:37.950329 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:37.950600 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:37.953668 (+ 3068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:37.953674 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:37.953731 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:37.954922 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:37.954939 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.957604 (+ 2665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.957611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:37.960037 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:37.960048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.963286 (+ 3238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.963293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:37.963487 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:37.963523 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 11:31:37.963628 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 11:31:37.963775 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":481,"spinlock_wait_cycles":2304,"thread_start_us":765,"threads_started":1}
I20260929 11:31:37.966131 9379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:37.949628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.23.153.3:46817)
0929 11:31:37.953006 (+ 3378us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:37.953014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:37.953050 (+ 36us) server_negotiation.cc:407] Connection header received
0929 11:31:37.953175 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:37.953184 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:37.953452 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:37.954078 (+ 626us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:37.955187 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.957397 (+ 2210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.960251 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.963043 (+ 2792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.963210 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:37.965518 (+ 2308us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:37.965626 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 11:31:37.965787 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3219,"thread_start_us":2806,"threads_started":1}
I20260929 11:31:37.967505 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:37.954017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39445 (local address 127.0.0.1:49306)
0929 11:31:37.954964 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:37.954991 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:37.958122 (+ 3131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:37.959486 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:37.959492 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:37.959526 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:37.959960 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:37.959973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.961970 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.961976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:37.964191 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:37.964211 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.966735 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.966742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:37.966936 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:37.966972 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 11:31:37.967103 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 11:31:37.967225 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":442,"threads_started":1}
I20260929 11:31:37.967267 9164 tablet_service.cc:1467] Processing CreateTablet for tablet 91c83686921c4eddbc035230d0048733 (DEFAULT_TABLE table=test-table [id=4c5395fe3ae74d4aba9e2860d31050d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:31:37.968324 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:37.953455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49306 (local address 127.23.153.1:39445)
0929 11:31:37.958438 (+ 4983us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:37.958449 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:37.958481 (+ 32us) server_negotiation.cc:407] Connection header received
0929 11:31:37.958627 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:37.958637 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:37.958962 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:37.959222 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:37.960192 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.961772 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.964430 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:37.965981 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:37.966133 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:37.967666 (+ 1533us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:37.967782 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 11:31:37.967987 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4777,"thread_start_us":4599,"threads_started":1}
I20260929 11:31:37.969087 9164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c83686921c4eddbc035230d0048733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:37.970571 9298 tablet_service.cc:1467] Processing CreateTablet for tablet 91c83686921c4eddbc035230d0048733 (DEFAULT_TABLE table=test-table [id=4c5395fe3ae74d4aba9e2860d31050d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:31:37.972297 9298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c83686921c4eddbc035230d0048733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:37.972373 9030 tablet_service.cc:1467] Processing CreateTablet for tablet 91c83686921c4eddbc035230d0048733 (DEFAULT_TABLE table=test-table [id=4c5395fe3ae74d4aba9e2860d31050d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:31:37.973927 9030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c83686921c4eddbc035230d0048733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:37.981662 9382 tablet_bootstrap.cc:492] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e: Bootstrap starting.
I20260929 11:31:37.986048 9382 tablet_bootstrap.cc:654] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:37.987224 9382 log.cc:824] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:37.989421 9385 tablet_bootstrap.cc:492] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673: Bootstrap starting.
I20260929 11:31:37.990298 9384 tablet_bootstrap.cc:492] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0: Bootstrap starting.
I20260929 11:31:37.990545 9382 tablet_bootstrap.cc:492] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e: No bootstrap required, opened a new log
I20260929 11:31:37.990850 9382 ts_tablet_manager.cc:1397] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 11:31:37.993505 9384 tablet_bootstrap.cc:654] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:37.993616 9385 tablet_bootstrap.cc:654] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:37.995173 9384 log.cc:824] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:37.996524 9385 log.cc:824] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:38.001096 9385 tablet_bootstrap.cc:492] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673: No bootstrap required, opened a new log
I20260929 11:31:38.001452 9385 ts_tablet_manager.cc:1397] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260929 11:31:38.001513 9384 tablet_bootstrap.cc:492] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0: No bootstrap required, opened a new log
I20260929 11:31:38.001818 9384 ts_tablet_manager.cc:1397] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260929 11:31:38.002429 9382 raft_consensus.cc:348] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600f7340b94842268b54b56d605755d0" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46817 } } peers { permanent_uuid: "a2e163380cda4c23a35152aa5312346e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39863 } } peers { permanent_uuid: "1f1daa4acf1147fc9c632be4bed43673" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39445 } }
I20260929 11:31:38.002952 9382 raft_consensus.cc:374] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:38.003099 9382 raft_consensus.cc:729] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e163380cda4c23a35152aa5312346e, State: Initialized, Role: FOLLOWER
I20260929 11:31:38.003827 9382 consensus_queue.cc:260] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600f7340b94842268b54b56d605755d0" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46817 } } peers { permanent_uuid: "a2e163380cda4c23a35152aa5312346e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39863 } } peers { permanent_uuid: "1f1daa4acf1147fc9c632be4bed43673" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39445 } }
I20260929 11:31:38.011457 9229 heartbeater.cc:503] Master 127.23.153.62:42487 was elected leader, sending a full tablet report...
I20260929 11:31:38.011258 9385 raft_consensus.cc:348] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600f7340b94842268b54b56d605755d0" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46817 } } peers { permanent_uuid: "a2e163380cda4c23a35152aa5312346e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39863 } } peers { permanent_uuid: "1f1daa4acf1147fc9c632be4bed43673" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39445 } }
I20260929 11:31:38.011787 9385 raft_consensus.cc:374] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:38.011886 9385 raft_consensus.cc:729] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f1daa4acf1147fc9c632be4bed43673, State: Initialized, Role: FOLLOWER
I20260929 11:31:38.012591 9382 ts_tablet_manager.cc:1428] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e: Time spent starting tablet: real 0.022s user 0.012s sys 0.008s
I20260929 11:31:38.012758 9385 consensus_queue.cc:260] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600f7340b94842268b54b56d605755d0" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46817 } } peers { permanent_uuid: "a2e163380cda4c23a35152aa5312346e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39863 } } peers { permanent_uuid: "1f1daa4acf1147fc9c632be4bed43673" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39445 } }
I20260929 11:31:38.013955 9384 raft_consensus.cc:348] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600f7340b94842268b54b56d605755d0" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46817 } } peers { permanent_uuid: "a2e163380cda4c23a35152aa5312346e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39863 } } peers { permanent_uuid: "1f1daa4acf1147fc9c632be4bed43673" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39445 } }
I20260929 11:31:38.014612 9384 raft_consensus.cc:374] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:38.014776 9384 raft_consensus.cc:729] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600f7340b94842268b54b56d605755d0, State: Initialized, Role: FOLLOWER
I20260929 11:31:38.015352 9385 ts_tablet_manager.cc:1428] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260929 11:31:38.015753 9384 consensus_queue.cc:260] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600f7340b94842268b54b56d605755d0" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46817 } } peers { permanent_uuid: "a2e163380cda4c23a35152aa5312346e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39863 } } peers { permanent_uuid: "1f1daa4acf1147fc9c632be4bed43673" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39445 } }
I20260929 11:31:38.016899 9363 heartbeater.cc:503] Master 127.23.153.62:42487 was elected leader, sending a full tablet report...
I20260929 11:31:38.017776 9384 ts_tablet_manager.cc:1428] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260929 11:31:38.056209 9230 tablet.cc:2369] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:31:38.124821 9364 tablet.cc:2369] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:31:38.166944 9388 raft_consensus.cc:482] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:31:38.167326 9388 raft_consensus.cc:504] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600f7340b94842268b54b56d605755d0" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46817 } } peers { permanent_uuid: "a2e163380cda4c23a35152aa5312346e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39863 } } peers { permanent_uuid: "1f1daa4acf1147fc9c632be4bed43673" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39445 } }
I20260929 11:31:38.169261 9388 leader_election.cc:288] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 600f7340b94842268b54b56d605755d0 (127.23.153.3:46817), 1f1daa4acf1147fc9c632be4bed43673 (127.23.153.1:39445)
I20260929 11:31:38.176808 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:38.170055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:48163 (local address 127.23.153.1:39445)
0929 11:31:38.170268 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:38.170274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:38.170675 (+ 401us) server_negotiation.cc:407] Connection header received
0929 11:31:38.170922 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:38.170926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:38.170993 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:38.171106 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:38.171833 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:38.172637 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:38.174742 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:38.175781 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:38.175976 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:38.176053 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:38.176390 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 11:31:38.176541 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 11:31:38.176826 9379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:38.169679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:40115 (local address 127.23.153.3:46817)
0929 11:31:38.169935 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:38.169943 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:38.170852 (+ 909us) server_negotiation.cc:407] Connection header received
0929 11:31:38.171065 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:38.171070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:38.171167 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:38.171272 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:38.171899 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:38.172661 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:38.174816 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:38.175710 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:38.175875 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:38.175959 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:38.176434 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 11:31:38.176574 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 11:31:38.177011 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:38.169642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39445 (local address 127.23.153.2:48163)
0929 11:31:38.170515 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:38.170543 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:38.170756 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:38.171135 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:38.171140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:38.171164 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:38.171655 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:38.171669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:38.172796 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:38.172802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:38.174529 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:38.174543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:38.176039 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:38.176045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:38.176202 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:38.176242 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 11:31:38.176324 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 11:31:38.176473 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":35,"spinlock_wait_cycles":30080,"thread_start_us":760,"threads_started":1}
I20260929 11:31:38.177054 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:38.169678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:46817 (local address 127.23.153.2:40115)
0929 11:31:38.170741 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:38.170755 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:38.170893 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:38.171306 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:38.171312 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:38.171344 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:38.171723 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:38.171735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:38.172808 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:38.172812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:38.174515 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:38.174534 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:38.175996 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:38.176003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:38.176255 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:38.176282 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 11:31:38.176379 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 11:31:38.176521 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"mutex_wait_us":71,"thread_start_us":446,"threads_started":1}
I20260929 11:31:38.178212 9318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c83686921c4eddbc035230d0048733" candidate_uuid: "a2e163380cda4c23a35152aa5312346e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "600f7340b94842268b54b56d605755d0" is_pre_election: true
I20260929 11:31:38.178205 9050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c83686921c4eddbc035230d0048733" candidate_uuid: "a2e163380cda4c23a35152aa5312346e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f1daa4acf1147fc9c632be4bed43673" is_pre_election: true
I20260929 11:31:38.178711 9050 raft_consensus.cc:2454] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e163380cda4c23a35152aa5312346e in term 0.
I20260929 11:31:38.178714 9318 raft_consensus.cc:2454] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2e163380cda4c23a35152aa5312346e in term 0.
I20260929 11:31:38.179483 9120 leader_election.cc:302] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 600f7340b94842268b54b56d605755d0, a2e163380cda4c23a35152aa5312346e; no voters:
I20260929 11:31:38.179913 9388 raft_consensus.cc:2781] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:31:38.180055 9388 raft_consensus.cc:482] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:31:38.180158 9388 raft_consensus.cc:3037] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:38.182545 9388 raft_consensus.cc:504] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600f7340b94842268b54b56d605755d0" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46817 } } peers { permanent_uuid: "a2e163380cda4c23a35152aa5312346e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39863 } } peers { permanent_uuid: "1f1daa4acf1147fc9c632be4bed43673" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39445 } }
I20260929 11:31:38.183362 9388 leader_election.cc:288] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [CANDIDATE]: Term 1 election: Requested vote from peers 600f7340b94842268b54b56d605755d0 (127.23.153.3:46817), 1f1daa4acf1147fc9c632be4bed43673 (127.23.153.1:39445)
I20260929 11:31:38.183884 9318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c83686921c4eddbc035230d0048733" candidate_uuid: "a2e163380cda4c23a35152aa5312346e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "600f7340b94842268b54b56d605755d0"
I20260929 11:31:38.184099 9318 raft_consensus.cc:3037] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:38.184078 9050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c83686921c4eddbc035230d0048733" candidate_uuid: "a2e163380cda4c23a35152aa5312346e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f1daa4acf1147fc9c632be4bed43673"
I20260929 11:31:38.184288 9050 raft_consensus.cc:3037] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:38.186903 9050 raft_consensus.cc:2454] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e163380cda4c23a35152aa5312346e in term 1.
I20260929 11:31:38.187381 9318 raft_consensus.cc:2454] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2e163380cda4c23a35152aa5312346e in term 1.
I20260929 11:31:38.187520 9118 leader_election.cc:302] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f1daa4acf1147fc9c632be4bed43673, a2e163380cda4c23a35152aa5312346e; no voters:
I20260929 11:31:38.187932 9388 raft_consensus.cc:2781] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:31:38.188457 9388 raft_consensus.cc:686] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [term 1 LEADER]: Becoming Leader. State: Replica: a2e163380cda4c23a35152aa5312346e, State: Running, Role: LEADER
I20260929 11:31:38.188999 9388 consensus_queue.cc:237] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600f7340b94842268b54b56d605755d0" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46817 } } peers { permanent_uuid: "a2e163380cda4c23a35152aa5312346e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39863 } } peers { permanent_uuid: "1f1daa4acf1147fc9c632be4bed43673" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39445 } }
I20260929 11:31:38.194794 8903 catalog_manager.cc:5495] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e reported cstate change: term changed from 0 to 1, leader changed from <none> to a2e163380cda4c23a35152aa5312346e (127.23.153.2). New cstate: current_term: 1 leader_uuid: "a2e163380cda4c23a35152aa5312346e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "600f7340b94842268b54b56d605755d0" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e163380cda4c23a35152aa5312346e" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f1daa4acf1147fc9c632be4bed43673" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39445 } health_report { overall_health: UNKNOWN } } }
W20260929 11:31:38.240540 9096 tablet.cc:2369] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:31:38.267948 9378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:38.262310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.23.153.2:39863)
0929 11:31:38.262487 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:38.262492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:38.262511 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:31:38.262700 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:38.262704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:38.262787 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:38.262931 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:31:38.263304 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:38.264037 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:38.265547 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:38.265982 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:38.266095 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:38.266698 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:31:38.267343 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:31:38.267465 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:38.267621 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 11:31:38.267715 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 11:31:38.277442 9050 raft_consensus.cc:1261] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673 [term 1 FOLLOWER]: Refusing update from remote peer a2e163380cda4c23a35152aa5312346e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:31:38.277793 9318 raft_consensus.cc:1261] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0 [term 1 FOLLOWER]: Refusing update from remote peer a2e163380cda4c23a35152aa5312346e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:31:38.278638 9393 consensus_queue.cc:1035] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f1daa4acf1147fc9c632be4bed43673" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:31:38.279078 9388 consensus_queue.cc:1035] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [LEADER]: Connected to new peer: Peer: permanent_uuid: "600f7340b94842268b54b56d605755d0" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:31:38.314779 8903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:31:38.315109 8903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:31:38.318789 8903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95eaf5f4fa46493e81f5f92ae0f9b050: Sending DeleteTablet for 3 replicas of tablet 91c83686921c4eddbc035230d0048733
I20260929 11:31:38.320145 9164 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c83686921c4eddbc035230d0048733 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:38 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:55642
I20260929 11:31:38.320727 9030 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c83686921c4eddbc035230d0048733 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:38 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:49306
I20260929 11:31:38.320955 9298 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c83686921c4eddbc035230d0048733 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:38 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:41500
I20260929 11:31:38.327391 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 8967
Sep 29 11:31:38 dist-test-slave-qk05 krb5kdc[8861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:31:38.347340 9408 tablet_replica.cc:330] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0: stopping tablet replica
I20260929 11:31:38.348063 9408 raft_consensus.cc:2229] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:31:38.348825 9408 raft_consensus.cc:2258] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:31:38.351212 9406 tablet_replica.cc:330] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e: stopping tablet replica
I20260929 11:31:38.352210 9406 raft_consensus.cc:2229] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:31:38.354548 9408 ts_tablet_manager.cc:1905] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:31:38.356010 9406 raft_consensus.cc:2258] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:31:38.356863 9407 tablet_replica.cc:330] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673: stopping tablet replica
I20260929 11:31:38.357370 9407 raft_consensus.cc:2229] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:31:38.358032 9407 raft_consensus.cc:2258] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:31:38.360199 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:38.328657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49316 (local address 127.23.153.1:39445)
0929 11:31:38.328836 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:38.328844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:38.333194 (+ 4350us) server_negotiation.cc:407] Connection header received
0929 11:31:38.333441 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:38.333448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:38.333584 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:38.333673 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:38.335747 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:38.337034 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:38.339252 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:38.340068 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:38.340182 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:38.349306 (+ 9124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:38.349334 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:38.349357 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:38.349424 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:38.354527 (+ 5103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:38.356240 (+ 1713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:38.356253 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:38.356256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:38.356522 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:38.357221 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:38.357227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:38.357230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:38.358016 (+ 786us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:38.358350 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:38.359523 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0929 11:31:38.359800 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 11:31:38.361685 9090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:38.362563 9406 ts_tablet_manager.cc:1905] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:31:38.365535 9408 ts_tablet_manager.cc:1918] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:31:38.365849 9408 log.cc:1196] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/91c83686921c4eddbc035230d0048733
I20260929 11:31:38.366478 9408 ts_tablet_manager.cc:1939] T 91c83686921c4eddbc035230d0048733 P 600f7340b94842268b54b56d605755d0: Deleting consensus metadata
I20260929 11:31:38.369256 8893 catalog_manager.cc:4841] TS 600f7340b94842268b54b56d605755d0 (127.23.153.3:46817): tablet 91c83686921c4eddbc035230d0048733 (table test-table [id=4c5395fe3ae74d4aba9e2860d31050d6]) successfully deleted
I20260929 11:31:38.373560 9406 ts_tablet_manager.cc:1918] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:31:38.373790 9406 log.cc:1196] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/91c83686921c4eddbc035230d0048733
I20260929 11:31:38.374629 9406 ts_tablet_manager.cc:1939] T 91c83686921c4eddbc035230d0048733 P a2e163380cda4c23a35152aa5312346e: Deleting consensus metadata
I20260929 11:31:38.377099 8891 catalog_manager.cc:4841] TS a2e163380cda4c23a35152aa5312346e (127.23.153.2:39863): tablet 91c83686921c4eddbc035230d0048733 (table test-table [id=4c5395fe3ae74d4aba9e2860d31050d6]) successfully deleted
I20260929 11:31:38.363224 9407 ts_tablet_manager.cc:1905] T 91c83686921c4eddbc035230d0048733 P 1f1daa4acf1147fc9c632be4bed43673: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:31:38.464385 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 8967
W20260929 11:31:38.488348 8891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.153.1:39445 (error 108)
I20260929 11:31:38.488600 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 9101
I20260929 11:31:38.490804 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:38.489912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39445 (local address 127.0.0.1:49322)
0929 11:31:38.490330 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:38.490521 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":208,"threads_started":1}
W20260929 11:31:38.491052 8891 catalog_manager.cc:4584] TS 1f1daa4acf1147fc9c632be4bed43673 (127.23.153.1:39445): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91c83686921c4eddbc035230d0048733: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39445: connect: Connection refused (error 111)
Sep 29 11:31:38 dist-test-slave-qk05 krb5kdc[8861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.2@KRBTEST.COM
I20260929 11:31:38.508237 9378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:38.489410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55666 (local address 127.23.153.2:39863)
0929 11:31:38.489605 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:38.489612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:38.489789 (+ 177us) server_negotiation.cc:407] Connection header received
0929 11:31:38.489990 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:38.489995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:38.490109 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:38.490228 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:38.492204 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:38.493419 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:38.495263 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:38.495962 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:38.496106 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:38.500492 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:38.500516 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:38.500539 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:38.500610 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:38.504886 (+ 4276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:38.505722 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:38.505732 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:38.505736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:38.505942 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:38.506358 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:38.506363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:38.506367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:38.506870 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:38.507372 (+ 502us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:38.507556 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 11:31:38.507819 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:31:38.509092 9224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:38.544953 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:38.544506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39445 (local address 127.0.0.1:49330)
0929 11:31:38.544724 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:38.544819 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 11:31:38.545948 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:38.545591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39445 (local address 127.0.0.1:49334)
0929 11:31:38.545737 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:38.545807 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 11:31:38.606478 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:38.605980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39445 (local address 127.0.0.1:49350)
0929 11:31:38.606228 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:38.606321 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 11:31:38.607667 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:38.607194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39445 (local address 127.0.0.1:49362)
0929 11:31:38.607418 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:38.607509 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 11:31:38.628628 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 9101
I20260929 11:31:38.645154 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 9235
Sep 29 11:31:38 dist-test-slave-qk05 krb5kdc[8861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.3@KRBTEST.COM
I20260929 11:31:38.662189 9379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:38.645923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.23.153.3:46817)
0929 11:31:38.646109 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:38.646113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:38.646148 (+ 35us) server_negotiation.cc:407] Connection header received
0929 11:31:38.646326 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:38.646331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:38.646406 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:38.646493 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:38.647976 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:38.648730 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:38.650179 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:38.650589 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:38.650745 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:38.655821 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:38.655840 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:38.655857 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:38.655904 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:38.658911 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:38.659889 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:38.659899 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:38.659902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:38.660096 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:38.660570 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:38.660575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:38.660577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:38.660998 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:38.661243 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:38.661668 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 11:31:38.661886 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 11:31:38.662931 9358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:38.688720 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:38.687958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39445 (local address 127.0.0.1:49364)
0929 11:31:38.688320 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:38.688492 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260929 11:31:38.690645 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:38.690000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39445 (local address 127.0.0.1:49370)
0929 11:31:38.690304 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:38.690443 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260929 11:31:38.754983 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 9235
I20260929 11:31:38.769533 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 8874
I20260929 11:31:38.770762 8936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:38.888301 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:38.887397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39445 (local address 127.0.0.1:49374)
0929 11:31:38.887999 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:38.888122 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452}
I20260929 11:31:38.888453 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 8874
2026-09-29T11:31:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4632 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T11:31:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:31:38Z Disabled control of system clock
I20260929 11:31:38.975831 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:38475
--webserver_interface=127.23.153.62
--webserver_port=0
--builtin_ntp_servers=127.23.153.20:36667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:38475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 11:31:39.348824 9419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:39.349171 9419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:39.349265 9419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:39.357960 9419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:31:39.358114 9419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:39.358152 9419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:39.358217 9419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:31:39.358263 9419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:31:39.369016 9419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:38475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:38475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:39.371204 9419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:39.373317 9419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:39.386168 9427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:39.386143 9424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:39.386255 9425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:39.387346 9419 server_base.cc:1034] running on GCE node
I20260929 11:31:39.388260 9419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:39.389945 9419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:39.391144 9419 hybrid_clock.cc:648] HybridClock initialized: now 1790681499391077 us; error 60 us; skew 500 ppm
I20260929 11:31:39.391541 9419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:31:39.400290 9419 webserver.cc:460] Webserver started at http://127.23.153.62:43969/ using document root <none> and password file <none>
I20260929 11:31:39.401087 9419 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:39.401201 9419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:39.401506 9419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:39.404807 9419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "085b1b90c0ab434891506b5ec752af98"
format_stamp: "Formatted at 2026-09-29 11:31:39 on dist-test-slave-qk05"
server_key: "77c9c1a284ba49e6ae6d967a5cd239fe"
server_key_iv: "cf88261bb9ccdf60c5634b9d3ab89367"
server_key_version: "encryptionkey@0"
I20260929 11:31:39.405683 9419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "085b1b90c0ab434891506b5ec752af98"
format_stamp: "Formatted at 2026-09-29 11:31:39 on dist-test-slave-qk05"
server_key: "77c9c1a284ba49e6ae6d967a5cd239fe"
server_key_iv: "cf88261bb9ccdf60c5634b9d3ab89367"
server_key_version: "encryptionkey@0"
I20260929 11:31:39.411590 9419 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 11:31:39.415836 9433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:39.417405 9419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:31:39.417613 9419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "085b1b90c0ab434891506b5ec752af98"
format_stamp: "Formatted at 2026-09-29 11:31:39 on dist-test-slave-qk05"
server_key: "77c9c1a284ba49e6ae6d967a5cd239fe"
server_key_iv: "cf88261bb9ccdf60c5634b9d3ab89367"
server_key_version: "encryptionkey@0"
I20260929 11:31:39.417850 9419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:39.459218 9419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:39.460155 9419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:39.460469 9419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:39.479307 9419 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:38475
I20260929 11:31:39.479315 9484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:38475 every 8 connection(s)
I20260929 11:31:39.480736 9419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:31:39.484553 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 9419
I20260929 11:31:39.484790 9485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:39.484818 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:31:39.485332 24164 external_mini_cluster.cc:1428] Setting key 5de3eb88ae9063cc8447bc5076f813d4
I20260929 11:31:39.500233 9485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98: Bootstrap starting.
I20260929 11:31:39.503090 9488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:39.487612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.23.153.62:38475)
0929 11:31:39.488792 (+ 1180us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:39.488807 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:39.488866 (+ 59us) server_negotiation.cc:407] Connection header received
0929 11:31:39.491662 (+ 2796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:39.491700 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:39.492463 (+ 763us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:39.492998 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:39.493598 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:39.495555 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:39.497127 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:39.498310 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:39.498504 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:39.499711 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:39.499726 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:39.499747 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:39.500482 (+ 735us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:39.500631 (+ 149us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:39.500661 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:39.500781 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:39.500966 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 11:31:39.501694 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":369,"threads_started":1}
I20260929 11:31:39.504740 9485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:39.505860 9485 log.cc:824] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:39.509259 9485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98: No bootstrap required, opened a new log
I20260929 11:31:39.517562 9485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085b1b90c0ab434891506b5ec752af98" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38475 } }
I20260929 11:31:39.517979 9485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:39.518056 9485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085b1b90c0ab434891506b5ec752af98, State: Initialized, Role: FOLLOWER
I20260929 11:31:39.518631 9485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085b1b90c0ab434891506b5ec752af98" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38475 } }
I20260929 11:31:39.518854 9485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:31:39.518960 9485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:31:39.519140 9485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:39.521363 9485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085b1b90c0ab434891506b5ec752af98" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38475 } }
I20260929 11:31:39.522013 9485 leader_election.cc:302] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 085b1b90c0ab434891506b5ec752af98; no voters:
I20260929 11:31:39.522847 9485 leader_election.cc:288] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:31:39.523036 9490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:31:39.524756 9490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98 [term 1 LEADER]: Becoming Leader. State: Replica: 085b1b90c0ab434891506b5ec752af98, State: Running, Role: LEADER
I20260929 11:31:39.525261 9485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:31:39.525326 9490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085b1b90c0ab434891506b5ec752af98" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38475 } }
I20260929 11:31:39.528609 9491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "085b1b90c0ab434891506b5ec752af98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085b1b90c0ab434891506b5ec752af98" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38475 } } }
I20260929 11:31:39.529390 9491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98 [sys.catalog]: This master's current role is: LEADER
I20260929 11:31:39.529016 9492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 085b1b90c0ab434891506b5ec752af98. Latest consensus state: current_term: 1 leader_uuid: "085b1b90c0ab434891506b5ec752af98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085b1b90c0ab434891506b5ec752af98" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 38475 } } }
I20260929 11:31:39.530499 9492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98 [sys.catalog]: This master's current role is: LEADER
I20260929 11:31:39.531796 9500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:31:39.538915 9500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:31:39.550184 9500 catalog_manager.cc:1347] Generated new cluster ID: 5b0654e47ff14169b3e55fd95aac37ae
I20260929 11:31:39.550319 9500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:31:39.568969 9500 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:31:39.570320 9500 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:31:39.580717 9500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98: Generated new TSK 0
I20260929 11:31:39.581941 9500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:31:39.599754 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38475
--builtin_ntp_servers=127.23.153.20:36667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 11:31:39.976572 9509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:39.976881 9509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:39.976985 9509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:39.985698 9509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:39.985917 9509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:39.986086 9509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:31:39.996115 9509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:39.997673 9509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:39.999711 9509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:40.019954 9515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:40.019963 9517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:40.019923 9514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:40.021220 9509 server_base.cc:1034] running on GCE node
I20260929 11:31:40.021931 9509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:40.023108 9509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:40.024375 9509 hybrid_clock.cc:648] HybridClock initialized: now 1790681500024309 us; error 58 us; skew 500 ppm
I20260929 11:31:40.025008 9509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:31:40.028288 9509 webserver.cc:460] Webserver started at http://127.23.153.1:33119/ using document root <none> and password file <none>
I20260929 11:31:40.029150 9509 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:40.029263 9509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:40.029589 9509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:40.032976 9509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "d21125d00561446cb8e55a05322c6a1a"
format_stamp: "Formatted at 2026-09-29 11:31:40 on dist-test-slave-qk05"
server_key: "7bca345804aa63974e6c93cac10cdd1c"
server_key_iv: "5642d4360ee85b3fa7b57578556b3bb5"
server_key_version: "encryptionkey@0"
I20260929 11:31:40.033929 9509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "d21125d00561446cb8e55a05322c6a1a"
format_stamp: "Formatted at 2026-09-29 11:31:40 on dist-test-slave-qk05"
server_key: "7bca345804aa63974e6c93cac10cdd1c"
server_key_iv: "5642d4360ee85b3fa7b57578556b3bb5"
server_key_version: "encryptionkey@0"
I20260929 11:31:40.041520 9509 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.008s
I20260929 11:31:40.045995 9523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:40.048094 9509 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 11:31:40.048312 9509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "d21125d00561446cb8e55a05322c6a1a"
format_stamp: "Formatted at 2026-09-29 11:31:40 on dist-test-slave-qk05"
server_key: "7bca345804aa63974e6c93cac10cdd1c"
server_key_iv: "5642d4360ee85b3fa7b57578556b3bb5"
server_key_version: "encryptionkey@0"
I20260929 11:31:40.048564 9509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:40.061892 9509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:40.062813 9509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:40.063210 9509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:40.064304 9509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:40.066169 9509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:40.066288 9509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:40.066398 9509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:40.066474 9509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:40.100800 9509 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:34659
I20260929 11:31:40.100857 9635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:34659 every 8 connection(s)
I20260929 11:31:40.102348 9509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:31:40.109803 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 9509
I20260929 11:31:40.109994 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:31:40.110517 24164 external_mini_cluster.cc:1428] Setting key 51e01e722e8049bd6446b9e0eb26f736
I20260929 11:31:40.113615 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38475
--builtin_ntp_servers=127.23.153.20:36667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 11:31:40.116297 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:40.105141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:47811 (local address 127.23.153.62:38475)
0929 11:31:40.105588 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:40.105592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:40.106827 (+ 1235us) server_negotiation.cc:407] Connection header received
0929 11:31:40.108710 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:40.108715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:40.108805 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:40.108901 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:40.110342 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:40.111221 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:40.113288 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:40.113747 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:40.113885 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:40.115409 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:40.115414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:40.115418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:40.115437 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:40.115477 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:40.115490 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:40.115589 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:40.115826 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 11:31:40.115938 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":225,"threads_started":1}
I20260929 11:31:40.117434 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:40.105704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:38475 (local address 127.23.153.1:47811)
0929 11:31:40.106626 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:40.106697 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:40.108400 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:40.109162 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:40.109172 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:40.109278 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:31:40.110116 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:40.110144 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:40.111418 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:40.111424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:40.113108 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:40.113126 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:40.114012 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:40.114032 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:40.114262 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:40.114983 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:31:40.114986 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:31:40.115109 (+ 123us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:31:40.115206 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:31:40.115605 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:31:40.115617 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:31:40.115625 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 11:31:40.115800 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 11:31:40.116247 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":274,"threads_started":1}
I20260929 11:31:40.119472 9636 heartbeater.cc:347] Connected to a master server at 127.23.153.62:38475
I20260929 11:31:40.119781 9636 heartbeater.cc:464] Registering TS with master...
I20260929 11:31:40.120563 9636 heartbeater.cc:511] Master 127.23.153.62:38475 requested a full tablet report, sending...
I20260929 11:31:40.122707 9450 ts_manager.cc:194] Registered new tserver with Master: d21125d00561446cb8e55a05322c6a1a (127.23.153.1:34659)
I20260929 11:31:40.124312 9450 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.153.1:47811
W20260929 11:31:40.470714 9640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:40.471083 9640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:40.471197 9640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:40.480620 9640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:40.480770 9640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:40.480909 9640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:31:40.492466 9640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:40.493873 9640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:40.496019 9640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:40.511194 9645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:40.511224 9648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:40.511257 9646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:40.512193 9640 server_base.cc:1034] running on GCE node
I20260929 11:31:40.513178 9640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:40.514761 9640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:40.516052 9640 hybrid_clock.cc:648] HybridClock initialized: now 1790681500515955 us; error 162 us; skew 500 ppm
I20260929 11:31:40.516433 9640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:31:40.519376 9640 webserver.cc:460] Webserver started at http://127.23.153.2:35089/ using document root <none> and password file <none>
I20260929 11:31:40.520176 9640 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:40.520295 9640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:40.520627 9640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:40.524091 9640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "6427939bfee640448e76246a673233a1"
format_stamp: "Formatted at 2026-09-29 11:31:40 on dist-test-slave-qk05"
server_key: "ac0ac05f04b0552e96958849d2a5992a"
server_key_iv: "f5eeddb6dedbfc63fb56216a7e785391"
server_key_version: "encryptionkey@0"
I20260929 11:31:40.525853 9640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "6427939bfee640448e76246a673233a1"
format_stamp: "Formatted at 2026-09-29 11:31:40 on dist-test-slave-qk05"
server_key: "ac0ac05f04b0552e96958849d2a5992a"
server_key_iv: "f5eeddb6dedbfc63fb56216a7e785391"
server_key_version: "encryptionkey@0"
I20260929 11:31:40.533036 9640 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260929 11:31:40.537289 9654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:40.539222 9640 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:31:40.539438 9640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "6427939bfee640448e76246a673233a1"
format_stamp: "Formatted at 2026-09-29 11:31:40 on dist-test-slave-qk05"
server_key: "ac0ac05f04b0552e96958849d2a5992a"
server_key_iv: "f5eeddb6dedbfc63fb56216a7e785391"
server_key_version: "encryptionkey@0"
I20260929 11:31:40.539683 9640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:40.563911 9640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:40.564823 9640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:40.565145 9640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:40.566238 9640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:40.567986 9640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:40.568099 9640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:40.568220 9640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:40.568293 9640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:40.605055 9640 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:45237
I20260929 11:31:40.605113 9766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:45237 every 8 connection(s)
I20260929 11:31:40.606562 9640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:31:40.611367 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 9640
I20260929 11:31:40.611558 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:31:40.612109 24164 external_mini_cluster.cc:1428] Setting key 8620ea752e9a7f04bcbfa263f88fb300
I20260929 11:31:40.615926 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38475
--builtin_ntp_servers=127.23.153.20:36667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 11:31:40.623600 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:40.609914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:39521 (local address 127.23.153.62:38475)
0929 11:31:40.610186 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:40.610192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:40.612695 (+ 2503us) server_negotiation.cc:407] Connection header received
0929 11:31:40.615199 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:40.615204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:40.615283 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:40.615364 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:40.617165 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:40.618083 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:40.620432 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:40.620897 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:40.621008 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:40.622758 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:40.622764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:40.622768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:40.622791 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:40.622828 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:40.622842 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:40.622920 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:40.623213 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 11:31:40.623343 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260929 11:31:40.625052 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:40.610864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:38475 (local address 127.23.153.2:39521)
0929 11:31:40.612461 (+ 1597us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:40.612531 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:40.614768 (+ 2237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:40.615796 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:40.615811 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:40.615982 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:31:40.616905 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:40.616950 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:40.618273 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:40.618279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:40.620219 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:40.620239 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:40.621163 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:40.621171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:40.621509 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:40.622332 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:31:40.622335 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:31:40.622394 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:31:40.622528 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:31:40.622946 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:31:40.622955 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:31:40.622964 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 11:31:40.623194 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 11:31:40.623684 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"thread_start_us":548,"threads_started":1}
I20260929 11:31:40.627151 9767 heartbeater.cc:347] Connected to a master server at 127.23.153.62:38475
I20260929 11:31:40.627490 9767 heartbeater.cc:464] Registering TS with master...
I20260929 11:31:40.628904 9767 heartbeater.cc:511] Master 127.23.153.62:38475 requested a full tablet report, sending...
I20260929 11:31:40.630831 9450 ts_manager.cc:194] Registered new tserver with Master: 6427939bfee640448e76246a673233a1 (127.23.153.2:45237)
I20260929 11:31:40.631963 9450 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.153.2:39521
W20260929 11:31:40.973155 9770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:40.973485 9770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:40.973582 9770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:40.981402 9770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:40.981549 9770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:40.981669 9770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:31:40.991741 9770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:38475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:40.993057 9770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:40.994973 9770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:41.010864 9776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:41.011083 9778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:41.011734 9775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:41.013440 9770 server_base.cc:1034] running on GCE node
I20260929 11:31:41.014225 9770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:41.015993 9770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:41.017306 9770 hybrid_clock.cc:648] HybridClock initialized: now 1790681501017225 us; error 76 us; skew 500 ppm
I20260929 11:31:41.017750 9770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:31:41.020534 9770 webserver.cc:460] Webserver started at http://127.23.153.3:41863/ using document root <none> and password file <none>
I20260929 11:31:41.021353 9770 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:41.021477 9770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:41.021788 9770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:41.025142 9770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "c01dd61d8e04418f8f2f83d28c90a237"
format_stamp: "Formatted at 2026-09-29 11:31:41 on dist-test-slave-qk05"
server_key: "b403d1c9a7deca47efbf3e6418629999"
server_key_iv: "ba5f28810787e455375aa6815f5a3f8d"
server_key_version: "encryptionkey@0"
I20260929 11:31:41.026305 9770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "c01dd61d8e04418f8f2f83d28c90a237"
format_stamp: "Formatted at 2026-09-29 11:31:41 on dist-test-slave-qk05"
server_key: "b403d1c9a7deca47efbf3e6418629999"
server_key_iv: "ba5f28810787e455375aa6815f5a3f8d"
server_key_version: "encryptionkey@0"
I20260929 11:31:41.032687 9770 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 11:31:41.037446 9784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:41.039214 9770 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:31:41.039428 9770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "c01dd61d8e04418f8f2f83d28c90a237"
format_stamp: "Formatted at 2026-09-29 11:31:41 on dist-test-slave-qk05"
server_key: "b403d1c9a7deca47efbf3e6418629999"
server_key_iv: "ba5f28810787e455375aa6815f5a3f8d"
server_key_version: "encryptionkey@0"
I20260929 11:31:41.039690 9770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:41.077766 9770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:41.078688 9770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:41.078980 9770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:41.080096 9770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:41.081772 9770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:41.081879 9770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:41.081995 9770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:41.082062 9770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:41.120082 9770 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:46639
I20260929 11:31:41.120160 9896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:46639 every 8 connection(s)
I20260929 11:31:41.121462 9770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:31:41.127317 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 9770
I20260929 11:31:41.127494 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:31:41.127733 9636 heartbeater.cc:503] Master 127.23.153.62:38475 was elected leader, sending a full tablet report...
I20260929 11:31:41.127945 24164 external_mini_cluster.cc:1428] Setting key 9e29fbe38df4e06dc595144e3248b3b3
I20260929 11:31:41.137523 9899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.124741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:58025 (local address 127.23.153.62:38475)
0929 11:31:41.125220 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:41.125225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:41.126852 (+ 1627us) server_negotiation.cc:407] Connection header received
0929 11:31:41.130177 (+ 3325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:41.130182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:41.130271 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:41.130368 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:41.131854 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.132643 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.134420 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.134850 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.134971 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:41.136555 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:41.136560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:41.136564 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:41.136584 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:41.136633 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:41.136647 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:41.136732 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:41.136994 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 11:31:41.137131 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":210,"threads_started":1}
I20260929 11:31:41.138609 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.125485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:38475 (local address 127.23.153.3:58025)
0929 11:31:41.126655 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:41.126701 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:41.129821 (+ 3120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:41.130700 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:41.130715 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:41.130850 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:31:41.131639 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:41.131664 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.132836 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.132841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:41.134228 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:41.134246 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.135141 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.135150 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:41.135401 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:41.136155 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:31:41.136158 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:31:41.136206 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:31:41.136311 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:31:41.136751 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:31:41.136762 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:31:41.136772 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 11:31:41.136946 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 11:31:41.137423 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":306,"threads_started":1}
I20260929 11:31:41.140153 9897 heartbeater.cc:347] Connected to a master server at 127.23.153.62:38475
I20260929 11:31:41.140447 9897 heartbeater.cc:464] Registering TS with master...
I20260929 11:31:41.141307 9897 heartbeater.cc:511] Master 127.23.153.62:38475 requested a full tablet report, sending...
I20260929 11:31:41.143091 9450 ts_manager.cc:194] Registered new tserver with Master: c01dd61d8e04418f8f2f83d28c90a237 (127.23.153.3:46639)
I20260929 11:31:41.144173 9450 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.153.3:58025
I20260929 11:31:41.144311 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:31:41.157315 9899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.149518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58310 (local address 127.23.153.62:38475)
0929 11:31:41.149826 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:41.149833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:41.150512 (+ 679us) server_negotiation.cc:407] Connection header received
0929 11:31:41.150763 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:41.150769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:41.150856 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:41.150954 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:41.151560 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.152603 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.154620 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.155231 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.155370 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:41.156560 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:41.156566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:41.156569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:41.156592 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:41.156629 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:41.156644 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:41.156728 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:41.156870 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 11:31:41.156989 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:31:41.162387 9450 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:31:41.170243 9450 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:31:41.196622 9911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.183633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:46639 (local address 127.0.0.1:36550)
0929 11:31:41.185276 (+ 1643us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:41.185336 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:41.185551 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:41.187634 (+ 2083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:41.187643 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:41.187679 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:41.188081 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:41.188097 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.190450 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.190455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:41.192926 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:41.192972 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.195594 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.195602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:41.195805 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:41.195899 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 11:31:41.195996 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 11:31:41.196234 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1306,"mutex_wait_us":56,"thread_start_us":3073,"threads_started":1}
I20260929 11:31:41.196681 9913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.184580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.23.153.1:34659)
0929 11:31:41.186559 (+ 1979us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:41.186566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:41.187177 (+ 611us) server_negotiation.cc:407] Connection header received
0929 11:31:41.187487 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:41.187495 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:41.187736 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:41.187905 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:41.188778 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.190555 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.193203 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.194829 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.194994 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:41.195341 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:41.196091 (+ 750us) server_negotiation.cc:299] Negotiation successful
0929 11:31:41.196293 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1804,"thread_start_us":1666,"threads_started":1}
I20260929 11:31:41.196995 9914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.187836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34196 (local address 127.23.153.2:45237)
0929 11:31:41.188425 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:41.188434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:41.189164 (+ 730us) server_negotiation.cc:407] Connection header received
0929 11:31:41.189328 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:41.189336 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:41.189598 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:41.189824 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:41.190595 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.192194 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.194315 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.195546 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.195683 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:41.195924 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:41.196464 (+ 540us) server_negotiation.cc:299] Negotiation successful
0929 11:31:41.196655 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":286,"threads_started":1}
I20260929 11:31:41.197605 9915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.188458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:45237 (local address 127.0.0.1:34196)
0929 11:31:41.189032 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:41.189051 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:41.189182 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:41.189966 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:41.189971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:41.190001 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:41.190382 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:41.190393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.192360 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.192366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:41.194091 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:41.194106 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.196069 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.196075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:41.196262 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:41.196291 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 11:31:41.197243 (+ 952us) client_negotiation.cc:240] Negotiation successful
0929 11:31:41.197355 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":291,"threads_started":1}
I20260929 11:31:41.197634 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.183989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36550 (local address 127.23.153.3:46639)
0929 11:31:41.186237 (+ 2248us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:41.186248 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:41.186287 (+ 39us) server_negotiation.cc:407] Connection header received
0929 11:31:41.186403 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:41.186411 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:41.187371 (+ 960us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:41.188313 (+ 942us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:41.188609 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.190287 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.193263 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.194829 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.194994 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:41.195342 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:41.197042 (+ 1700us) server_negotiation.cc:299] Negotiation successful
0929 11:31:41.197248 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2057,"thread_start_us":1893,"threads_started":1}
I20260929 11:31:41.197671 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.183907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:34659 (local address 127.0.0.1:49668)
0929 11:31:41.187060 (+ 3153us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:41.187077 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:41.187283 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:41.188179 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:41.188184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:41.188211 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:41.188551 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:41.188563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.190722 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.190727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:41.192947 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:41.192964 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.195504 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.195511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:41.195773 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:41.195852 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 11:31:41.196571 (+ 719us) client_negotiation.cc:240] Negotiation successful
0929 11:31:41.197293 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2983,"mutex_wait_us":27,"thread_start_us":881,"threads_started":1}
I20260929 11:31:41.201694 9702 tablet_service.cc:1467] Processing CreateTablet for tablet cdb0b2cf371f4e67b626ca2548ea958f (DEFAULT_TABLE table=test-table [id=4ab8755a51444a47a2ba51b153fd2437]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:31:41.201696 9571 tablet_service.cc:1467] Processing CreateTablet for tablet cdb0b2cf371f4e67b626ca2548ea958f (DEFAULT_TABLE table=test-table [id=4ab8755a51444a47a2ba51b153fd2437]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:31:41.201691 9832 tablet_service.cc:1467] Processing CreateTablet for tablet cdb0b2cf371f4e67b626ca2548ea958f (DEFAULT_TABLE table=test-table [id=4ab8755a51444a47a2ba51b153fd2437]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:31:41.203437 9571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb0b2cf371f4e67b626ca2548ea958f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:41.203437 9832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb0b2cf371f4e67b626ca2548ea958f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:41.203439 9702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb0b2cf371f4e67b626ca2548ea958f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:41.219118 9916 tablet_bootstrap.cc:492] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1: Bootstrap starting.
I20260929 11:31:41.221593 9917 tablet_bootstrap.cc:492] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a: Bootstrap starting.
I20260929 11:31:41.222507 9918 tablet_bootstrap.cc:492] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237: Bootstrap starting.
I20260929 11:31:41.223654 9916 tablet_bootstrap.cc:654] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:41.226497 9917 tablet_bootstrap.cc:654] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:41.226509 9918 tablet_bootstrap.cc:654] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:41.226634 9916 log.cc:824] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:41.227852 9918 log.cc:824] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:41.227890 9917 log.cc:824] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:41.230279 9916 tablet_bootstrap.cc:492] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1: No bootstrap required, opened a new log
I20260929 11:31:41.230613 9916 ts_tablet_manager.cc:1397] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260929 11:31:41.230842 9918 tablet_bootstrap.cc:492] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237: No bootstrap required, opened a new log
I20260929 11:31:41.231005 9917 tablet_bootstrap.cc:492] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a: No bootstrap required, opened a new log
I20260929 11:31:41.231151 9918 ts_tablet_manager.cc:1397] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260929 11:31:41.231329 9917 ts_tablet_manager.cc:1397] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 11:31:41.239094 9918 raft_consensus.cc:348] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01dd61d8e04418f8f2f83d28c90a237" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46639 } } peers { permanent_uuid: "d21125d00561446cb8e55a05322c6a1a" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34659 } } peers { permanent_uuid: "6427939bfee640448e76246a673233a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45237 } }
I20260929 11:31:41.239526 9918 raft_consensus.cc:374] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:41.239626 9918 raft_consensus.cc:729] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c01dd61d8e04418f8f2f83d28c90a237, State: Initialized, Role: FOLLOWER
I20260929 11:31:41.240417 9918 consensus_queue.cc:260] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01dd61d8e04418f8f2f83d28c90a237" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46639 } } peers { permanent_uuid: "d21125d00561446cb8e55a05322c6a1a" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34659 } } peers { permanent_uuid: "6427939bfee640448e76246a673233a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45237 } }
I20260929 11:31:41.241648 9897 heartbeater.cc:503] Master 127.23.153.62:38475 was elected leader, sending a full tablet report...
I20260929 11:31:41.241578 9916 raft_consensus.cc:348] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01dd61d8e04418f8f2f83d28c90a237" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46639 } } peers { permanent_uuid: "d21125d00561446cb8e55a05322c6a1a" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34659 } } peers { permanent_uuid: "6427939bfee640448e76246a673233a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45237 } }
I20260929 11:31:41.242067 9916 raft_consensus.cc:374] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:41.242184 9916 raft_consensus.cc:729] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6427939bfee640448e76246a673233a1, State: Initialized, Role: FOLLOWER
I20260929 11:31:41.242261 9918 ts_tablet_manager.cc:1428] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260929 11:31:41.242844 9916 consensus_queue.cc:260] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01dd61d8e04418f8f2f83d28c90a237" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46639 } } peers { permanent_uuid: "d21125d00561446cb8e55a05322c6a1a" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34659 } } peers { permanent_uuid: "6427939bfee640448e76246a673233a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45237 } }
I20260929 11:31:41.243480 9917 raft_consensus.cc:348] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01dd61d8e04418f8f2f83d28c90a237" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46639 } } peers { permanent_uuid: "d21125d00561446cb8e55a05322c6a1a" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34659 } } peers { permanent_uuid: "6427939bfee640448e76246a673233a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45237 } }
I20260929 11:31:41.243984 9917 raft_consensus.cc:374] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:41.244079 9917 raft_consensus.cc:729] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d21125d00561446cb8e55a05322c6a1a, State: Initialized, Role: FOLLOWER
I20260929 11:31:41.244252 9767 heartbeater.cc:503] Master 127.23.153.62:38475 was elected leader, sending a full tablet report...
I20260929 11:31:41.244833 9916 ts_tablet_manager.cc:1428] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260929 11:31:41.244709 9917 consensus_queue.cc:260] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01dd61d8e04418f8f2f83d28c90a237" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46639 } } peers { permanent_uuid: "d21125d00561446cb8e55a05322c6a1a" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34659 } } peers { permanent_uuid: "6427939bfee640448e76246a673233a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45237 } }
I20260929 11:31:41.246719 9917 ts_tablet_manager.cc:1428] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260929 11:31:41.354967 9637 tablet.cc:2369] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:31:41.358829 9768 tablet.cc:2369] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 11:31:41.373495 9898 tablet.cc:2369] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:31:41.554239 9924 raft_consensus.cc:482] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:31:41.554589 9924 raft_consensus.cc:504] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01dd61d8e04418f8f2f83d28c90a237" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46639 } } peers { permanent_uuid: "d21125d00561446cb8e55a05322c6a1a" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34659 } } peers { permanent_uuid: "6427939bfee640448e76246a673233a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45237 } }
I20260929 11:31:41.556469 9924 leader_election.cc:288] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c01dd61d8e04418f8f2f83d28c90a237 (127.23.153.3:46639), 6427939bfee640448e76246a673233a1 (127.23.153.2:45237)
I20260929 11:31:41.562660 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.556992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:43373 (local address 127.23.153.3:46639)
0929 11:31:41.557170 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:41.557175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:41.557439 (+ 264us) server_negotiation.cc:407] Connection header received
0929 11:31:41.557681 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:41.557686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:41.557773 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:41.557884 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:41.558448 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.559227 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.561041 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.561792 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.561962 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:41.562040 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:41.562295 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 11:31:41.562425 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 11:31:41.562726 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.556722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:46639 (local address 127.23.153.1:43373)
0929 11:31:41.557333 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:41.557353 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:41.557504 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:41.557933 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:41.557938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:41.557969 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:41.558295 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:41.558305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.559358 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.559362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:41.560835 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:41.560851 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.561981 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.561988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:41.562142 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:41.562166 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 11:31:41.562247 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 11:31:41.562384 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":298,"threads_started":1}
I20260929 11:31:41.563127 9914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.557617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:56429 (local address 127.23.153.2:45237)
0929 11:31:41.557825 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:41.557831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:41.558160 (+ 329us) server_negotiation.cc:407] Connection header received
0929 11:31:41.558345 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:41.558350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:41.558425 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:41.558502 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:41.559073 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.559751 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.561471 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.562225 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.562375 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:41.562443 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:41.562729 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 11:31:41.562858 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:31:41.563143 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.557476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:45237 (local address 127.23.153.1:56429)
0929 11:31:41.558048 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:41.558064 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:41.558206 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:41.558535 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:41.558540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:41.558566 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:41.558887 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:41.558896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.559891 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.559900 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:41.561281 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:41.561295 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.562415 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.562422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:41.562574 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:41.562598 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 11:31:41.562672 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 11:31:41.562806 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":276,"threads_started":1}
I20260929 11:31:41.563927 9852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb0b2cf371f4e67b626ca2548ea958f" candidate_uuid: "d21125d00561446cb8e55a05322c6a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c01dd61d8e04418f8f2f83d28c90a237" is_pre_election: true
I20260929 11:31:41.563947 9722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb0b2cf371f4e67b626ca2548ea958f" candidate_uuid: "d21125d00561446cb8e55a05322c6a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6427939bfee640448e76246a673233a1" is_pre_election: true
I20260929 11:31:41.564409 9852 raft_consensus.cc:2454] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d21125d00561446cb8e55a05322c6a1a in term 0.
I20260929 11:31:41.564409 9722 raft_consensus.cc:2454] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d21125d00561446cb8e55a05322c6a1a in term 0.
I20260929 11:31:41.565037 9524 leader_election.cc:302] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6427939bfee640448e76246a673233a1, d21125d00561446cb8e55a05322c6a1a; no voters:
I20260929 11:31:41.565459 9924 raft_consensus.cc:2781] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:31:41.565619 9924 raft_consensus.cc:482] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:31:41.565718 9924 raft_consensus.cc:3037] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:41.568055 9924 raft_consensus.cc:504] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01dd61d8e04418f8f2f83d28c90a237" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46639 } } peers { permanent_uuid: "d21125d00561446cb8e55a05322c6a1a" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34659 } } peers { permanent_uuid: "6427939bfee640448e76246a673233a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45237 } }
I20260929 11:31:41.568694 9924 leader_election.cc:288] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [CANDIDATE]: Term 1 election: Requested vote from peers c01dd61d8e04418f8f2f83d28c90a237 (127.23.153.3:46639), 6427939bfee640448e76246a673233a1 (127.23.153.2:45237)
I20260929 11:31:41.569326 9852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb0b2cf371f4e67b626ca2548ea958f" candidate_uuid: "d21125d00561446cb8e55a05322c6a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c01dd61d8e04418f8f2f83d28c90a237"
I20260929 11:31:41.569350 9722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb0b2cf371f4e67b626ca2548ea958f" candidate_uuid: "d21125d00561446cb8e55a05322c6a1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6427939bfee640448e76246a673233a1"
I20260929 11:31:41.569535 9852 raft_consensus.cc:3037] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:41.569533 9722 raft_consensus.cc:3037] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:41.571938 9852 raft_consensus.cc:2454] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d21125d00561446cb8e55a05322c6a1a in term 1.
I20260929 11:31:41.572098 9722 raft_consensus.cc:2454] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d21125d00561446cb8e55a05322c6a1a in term 1.
I20260929 11:31:41.572444 9524 leader_election.cc:302] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c01dd61d8e04418f8f2f83d28c90a237, d21125d00561446cb8e55a05322c6a1a; no voters:
I20260929 11:31:41.572767 9924 raft_consensus.cc:2781] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:31:41.573189 9924 raft_consensus.cc:686] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [term 1 LEADER]: Becoming Leader. State: Replica: d21125d00561446cb8e55a05322c6a1a, State: Running, Role: LEADER
I20260929 11:31:41.573710 9924 consensus_queue.cc:237] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01dd61d8e04418f8f2f83d28c90a237" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46639 } } peers { permanent_uuid: "d21125d00561446cb8e55a05322c6a1a" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34659 } } peers { permanent_uuid: "6427939bfee640448e76246a673233a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45237 } }
I20260929 11:31:41.579101 9448 catalog_manager.cc:5495] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a reported cstate change: term changed from 0 to 1, leader changed from <none> to d21125d00561446cb8e55a05322c6a1a (127.23.153.1). New cstate: current_term: 1 leader_uuid: "d21125d00561446cb8e55a05322c6a1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c01dd61d8e04418f8f2f83d28c90a237" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d21125d00561446cb8e55a05322c6a1a" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 34659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6427939bfee640448e76246a673233a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45237 } health_report { overall_health: UNKNOWN } } }
I20260929 11:31:41.656692 9913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.650846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49672 (local address 127.23.153.1:34659)
0929 11:31:41.651094 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:41.651098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:41.651117 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:31:41.651270 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:41.651274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:41.651382 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:41.651529 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:31:41.651951 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.652688 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.654194 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.654644 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.654750 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:41.655353 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:31:41.656056 (+ 703us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:31:41.656195 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:41.656340 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 11:31:41.656421 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 11:31:41.665542 9722 raft_consensus.cc:1261] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1 [term 1 FOLLOWER]: Refusing update from remote peer d21125d00561446cb8e55a05322c6a1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:31:41.665544 9852 raft_consensus.cc:1261] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237 [term 1 FOLLOWER]: Refusing update from remote peer d21125d00561446cb8e55a05322c6a1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:31:41.666736 9924 consensus_queue.cc:1035] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6427939bfee640448e76246a673233a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:31:41.667112 9927 consensus_queue.cc:1035] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c01dd61d8e04418f8f2f83d28c90a237" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:31:41.679947 9930 mvcc.cc:205] Tried to move back new op lower bound from 7334631430811140096 to 7334631430449979392. Current Snapshot: MvccSnapshot[applied={T|T < 7334631430811140096}]
I20260929 11:31:41.684459 9931 mvcc.cc:205] Tried to move back new op lower bound from 7334631430811140096 to 7334631430449979392. Current Snapshot: MvccSnapshot[applied={T|T < 7334631430811140096}]
I20260929 11:31:41.696341 9449 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:31:41.696677 9449 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:31:41.701359 9449 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 085b1b90c0ab434891506b5ec752af98: Sending DeleteTablet for 3 replicas of tablet cdb0b2cf371f4e67b626ca2548ea958f
I20260929 11:31:41.703765 9702 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb0b2cf371f4e67b626ca2548ea958f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:41 UTC) from {username='slave'} at 127.0.0.1:34196
I20260929 11:31:41.705143 9832 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb0b2cf371f4e67b626ca2548ea958f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:41 UTC) from {username='slave'} at 127.0.0.1:36550
I20260929 11:31:41.706841 9571 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb0b2cf371f4e67b626ca2548ea958f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:41 UTC) from {username='slave'} at 127.0.0.1:49668
I20260929 11:31:41.708231 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 9509
I20260929 11:31:41.725592 9940 tablet_replica.cc:330] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237: stopping tablet replica
I20260929 11:31:41.727284 9940 raft_consensus.cc:2229] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:31:41.727911 9940 raft_consensus.cc:2258] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:31:41.728410 9939 tablet_replica.cc:330] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1: stopping tablet replica
I20260929 11:31:41.728886 9939 raft_consensus.cc:2229] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:31:41.729445 9939 raft_consensus.cc:2258] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:31:41.730240 9941 tablet_replica.cc:330] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a: stopping tablet replica
I20260929 11:31:41.730706 9941 raft_consensus.cc:2229] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:31:41.730744 9940 ts_tablet_manager.cc:1905] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:31:41.731482 9941 raft_consensus.cc:2258] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:31:41.733765 9913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.709737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49684 (local address 127.23.153.1:34659)
0929 11:31:41.709999 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:41.710009 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:41.720376 (+ 10367us) server_negotiation.cc:407] Connection header received
0929 11:31:41.721019 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:41.721027 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:41.721179 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:41.721269 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:41.722172 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.723661 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.725915 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.726757 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.726894 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:41.731699 (+ 4805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:41.731713 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:41.731735 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:41.732601 (+ 866us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:41.732672 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:41.732710 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:41.732818 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:41.733207 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 11:31:41.733341 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:31:41.735278 9631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:41.735857 9939 ts_tablet_manager.cc:1905] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:31:41.738911 9940 ts_tablet_manager.cc:1918] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:31:41.739156 9940 log.cc:1196] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/cdb0b2cf371f4e67b626ca2548ea958f
I20260929 11:31:41.739703 9940 ts_tablet_manager.cc:1939] T cdb0b2cf371f4e67b626ca2548ea958f P c01dd61d8e04418f8f2f83d28c90a237: Deleting consensus metadata
I20260929 11:31:41.741834 9434 catalog_manager.cc:4841] TS c01dd61d8e04418f8f2f83d28c90a237 (127.23.153.3:46639): tablet cdb0b2cf371f4e67b626ca2548ea958f (table test-table [id=4ab8755a51444a47a2ba51b153fd2437]) successfully deleted
I20260929 11:31:41.742905 9939 ts_tablet_manager.cc:1918] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:31:41.743134 9939 log.cc:1196] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/cdb0b2cf371f4e67b626ca2548ea958f
I20260929 11:31:41.743597 9939 ts_tablet_manager.cc:1939] T cdb0b2cf371f4e67b626ca2548ea958f P 6427939bfee640448e76246a673233a1: Deleting consensus metadata
I20260929 11:31:41.745661 9434 catalog_manager.cc:4841] TS 6427939bfee640448e76246a673233a1 (127.23.153.2:45237): tablet cdb0b2cf371f4e67b626ca2548ea958f (table test-table [id=4ab8755a51444a47a2ba51b153fd2437]) successfully deleted
I20260929 11:31:41.733350 9941 ts_tablet_manager.cc:1905] T cdb0b2cf371f4e67b626ca2548ea958f P d21125d00561446cb8e55a05322c6a1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:31:41.836568 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 9509
W20260929 11:31:41.858085 9435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.153.1:34659 (error 108)
I20260929 11:31:41.858244 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 9640
I20260929 11:31:41.860280 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.859454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:34659 (local address 127.0.0.1:49698)
0929 11:31:41.859874 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:41.860015 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:34659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"threads_started":1}
W20260929 11:31:41.860708 9435 catalog_manager.cc:4584] TS d21125d00561446cb8e55a05322c6a1a (127.23.153.1:34659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cdb0b2cf371f4e67b626ca2548ea958f: Network error: Client connection negotiation failed: client connection to 127.23.153.1:34659: connect: Connection refused (error 111)
I20260929 11:31:41.866506 9914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.859531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34210 (local address 127.23.153.2:45237)
0929 11:31:41.859732 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:41.859738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:41.859766 (+ 28us) server_negotiation.cc:407] Connection header received
0929 11:31:41.859849 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:41.859859 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:41.859989 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:41.860120 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:41.860804 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.861674 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.863071 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.863608 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.863735 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:41.864955 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:41.864964 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:41.864979 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:41.865621 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:41.865680 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:41.865701 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:41.865813 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:41.865986 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 11:31:41.866112 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 11:31:41.867268 9762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:41.914469 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.913943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:34659 (local address 127.0.0.1:49706)
0929 11:31:41.914207 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:41.914314 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:34659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260929 11:31:41.915632 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.915217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:34659 (local address 127.0.0.1:49716)
0929 11:31:41.915382 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:41.915463 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:34659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 11:31:41.960784 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 9640
I20260929 11:31:41.975612 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 9770
I20260929 11:31:41.976312 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.975835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:34659 (local address 127.0.0.1:49732)
0929 11:31:41.976029 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:41.976115 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:34659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 11:31:41.977516 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.977076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:34659 (local address 127.0.0.1:49748)
0929 11:31:41.977230 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:41.977321 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:34659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 11:31:41.982573 9912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:41.976376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.23.153.3:46639)
0929 11:31:41.976572 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:41.976577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:41.976600 (+ 23us) server_negotiation.cc:407] Connection header received
0929 11:31:41.976805 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:41.976814 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:41.976956 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:41.977073 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:41.977538 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.978289 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.979709 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:41.980116 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:41.980220 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:41.981204 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:41.981211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:41.981228 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:41.981836 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:41.981890 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:41.981912 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:41.982003 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:41.982129 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 11:31:41.982235 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:31:41.983310 9892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:42.058212 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:42.057719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:34659 (local address 127.0.0.1:49750)
0929 11:31:42.057960 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:42.058049 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:34659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 11:31:42.059458 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:42.058964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:34659 (local address 127.0.0.1:49766)
0929 11:31:42.059215 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:42.059296 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:34659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260929 11:31:42.076342 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 9770
I20260929 11:31:42.092217 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 9419
I20260929 11:31:42.093392 9480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:42.207810 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 9419
2026-09-29T11:31:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3315 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:31:42 dist-test-slave-qk05 krb5kdc[9951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:31:42 dist-test-slave-qk05 krb5kdc[9951](info): set up 2 sockets
Sep 29 11:31:42 dist-test-slave-qk05 krb5kdc[9951](info): commencing operation
krb5kdc: starting...
W20260929 11:31:43.267191 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:31:43 dist-test-slave-qk05 krb5kdc[9951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:31:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:31:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:31:43.523351 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:46565
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:46565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:31:43.893798 9964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:43.894156 9964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:43.894279 9964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:43.902349 9964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:31:43.902515 9964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:43.902599 9964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:43.902676 9964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:31:43.902737 9964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:31:43.913275 9964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:46565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:46565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:43.914681 9964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:43.916419 9964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:43.931691 9972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:43.931749 9970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:43.932251 9969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:43.933179 9964 server_base.cc:1034] running on GCE node
I20260929 11:31:43.934201 9964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:43.935890 9964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:43.937124 9964 hybrid_clock.cc:648] HybridClock initialized: now 1790681503937062 us; error 58 us; skew 500 ppm
I20260929 11:31:43.937474 9964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:31:43 dist-test-slave-qk05 krb5kdc[9951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681503, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:31:43.941313 9964 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:31:43.943326 9964 webserver.cc:460] Webserver started at http://127.23.153.62:43483/ using document root <none> and password file <none>
I20260929 11:31:43.944094 9964 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:43.944226 9964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:43.944543 9964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:43.947901 9964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "30df268f31cd43b99095af50e58fc1bf"
format_stamp: "Formatted at 2026-09-29 11:31:43 on dist-test-slave-qk05"
server_key: "5411d069fd328296ae64f1128249ee8a"
server_key_iv: "31674dbf3dca84c2b4044df546e2c94e"
server_key_version: "encryptionkey@0"
I20260929 11:31:43.948809 9964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "30df268f31cd43b99095af50e58fc1bf"
format_stamp: "Formatted at 2026-09-29 11:31:43 on dist-test-slave-qk05"
server_key: "5411d069fd328296ae64f1128249ee8a"
server_key_iv: "31674dbf3dca84c2b4044df546e2c94e"
server_key_version: "encryptionkey@0"
I20260929 11:31:43.954989 9964 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 11:31:43.959602 9979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:43.961335 9964 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260929 11:31:43.961552 9964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "30df268f31cd43b99095af50e58fc1bf"
format_stamp: "Formatted at 2026-09-29 11:31:43 on dist-test-slave-qk05"
server_key: "5411d069fd328296ae64f1128249ee8a"
server_key_iv: "31674dbf3dca84c2b4044df546e2c94e"
server_key_version: "encryptionkey@0"
I20260929 11:31:43.961804 9964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:43.983129 9964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:43.987581 9964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:43.987854 9964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:44.006449 9964 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:46565
I20260929 11:31:44.006484 10030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:46565 every 8 connection(s)
I20260929 11:31:44.008064 9964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:31:44.010993 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 9964
I20260929 11:31:44.011232 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:31:44.011792 24164 external_mini_cluster.cc:1428] Setting key 7e3bfa43d718a8bc844edb38a863c4a0
I20260929 11:31:44.013301 10031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 11:31:44 dist-test-slave-qk05 krb5kdc[9951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:31:44.029445 10031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf: Bootstrap starting.
I20260929 11:31:44.033947 10031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:44.035251 10031 log.cc:824] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:44.038962 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:44.013882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47634 (local address 127.23.153.62:46565)
0929 11:31:44.014702 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:44.014715 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:44.014759 (+ 44us) server_negotiation.cc:407] Connection header received
0929 11:31:44.016497 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:44.016523 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:44.017029 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:44.017427 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:44.019398 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:44.021546 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:44.023303 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:44.024453 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:44.024705 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:44.029457 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:44.029472 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:44.029495 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:44.029574 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:44.033659 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:44.034604 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:44.034614 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:44.034623 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:44.034863 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:44.035380 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:44.035385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:44.035388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:44.036179 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:44.036431 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:44.037055 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 11:31:44.037550 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":362,"threads_started":1}
I20260929 11:31:44.039150 10031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf: No bootstrap required, opened a new log
I20260929 11:31:44.047755 10031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30df268f31cd43b99095af50e58fc1bf" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 46565 } }
I20260929 11:31:44.048184 10031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:44.048257 10031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30df268f31cd43b99095af50e58fc1bf, State: Initialized, Role: FOLLOWER
I20260929 11:31:44.048866 10031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30df268f31cd43b99095af50e58fc1bf" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 46565 } }
I20260929 11:31:44.049099 10031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:31:44.049213 10031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:31:44.049350 10031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:44.051930 10031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30df268f31cd43b99095af50e58fc1bf" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 46565 } }
I20260929 11:31:44.052536 10031 leader_election.cc:302] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30df268f31cd43b99095af50e58fc1bf; no voters:
I20260929 11:31:44.053131 10031 leader_election.cc:288] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:31:44.053277 10036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:31:44.054152 10036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf [term 1 LEADER]: Becoming Leader. State: Replica: 30df268f31cd43b99095af50e58fc1bf, State: Running, Role: LEADER
I20260929 11:31:44.054976 10036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30df268f31cd43b99095af50e58fc1bf" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 46565 } }
I20260929 11:31:44.055312 10031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:31:44.059072 10038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30df268f31cd43b99095af50e58fc1bf. Latest consensus state: current_term: 1 leader_uuid: "30df268f31cd43b99095af50e58fc1bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30df268f31cd43b99095af50e58fc1bf" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 46565 } } }
I20260929 11:31:44.059043 10037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30df268f31cd43b99095af50e58fc1bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30df268f31cd43b99095af50e58fc1bf" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 46565 } } }
I20260929 11:31:44.059564 10038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf [sys.catalog]: This master's current role is: LEADER
I20260929 11:31:44.059566 10037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf [sys.catalog]: This master's current role is: LEADER
I20260929 11:31:44.064532 10045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:31:44.069674 10045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:31:44.081034 10045 catalog_manager.cc:1347] Generated new cluster ID: ddb1ad2604594a2c84016efc000c7733
I20260929 11:31:44.081195 10045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:31:44.103315 10045 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:31:44.104918 10045 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:31:44.114169 10045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf: Generated new TSK 0
I20260929 11:31:44.115442 10045 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
I20260929 11:31:44.178186 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:46565
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:31:44.543170 10057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:44.543485 10057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:44.543622 10057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:44.551630 10057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:44.551781 10057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:44.551951 10057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:31:44.563131 10057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:46565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:44.564549 10057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:44.566459 10057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:44.580880 10063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:44.580910 10062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:44.580847 10065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:44.582046 10057 server_base.cc:1034] running on GCE node
I20260929 11:31:44.583338 10057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:44.584529 10057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:44.585908 10057 hybrid_clock.cc:648] HybridClock initialized: now 1790681504585848 us; error 69 us; skew 500 ppm
I20260929 11:31:44.586309 10057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:31:44 dist-test-slave-qk05 krb5kdc[9951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681504, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:31:44.595541 10057 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:31:44.597649 10057 webserver.cc:460] Webserver started at http://127.23.153.1:32853/ using document root <none> and password file <none>
I20260929 11:31:44.598416 10057 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:44.598541 10057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:44.598843 10057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:44.602172 10057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "92445029d03a47289d829b5fc7205929"
format_stamp: "Formatted at 2026-09-29 11:31:44 on dist-test-slave-qk05"
server_key: "61102fcda896cd08225bab41e8d4f504"
server_key_iv: "fda09e71ab88db5b3a82f3f65d9c2027"
server_key_version: "encryptionkey@0"
I20260929 11:31:44.603171 10057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "92445029d03a47289d829b5fc7205929"
format_stamp: "Formatted at 2026-09-29 11:31:44 on dist-test-slave-qk05"
server_key: "61102fcda896cd08225bab41e8d4f504"
server_key_iv: "fda09e71ab88db5b3a82f3f65d9c2027"
server_key_version: "encryptionkey@0"
I20260929 11:31:44.611111 10057 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260929 11:31:44.616240 10072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:44.618002 10057 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260929 11:31:44.618204 10057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "92445029d03a47289d829b5fc7205929"
format_stamp: "Formatted at 2026-09-29 11:31:44 on dist-test-slave-qk05"
server_key: "61102fcda896cd08225bab41e8d4f504"
server_key_iv: "fda09e71ab88db5b3a82f3f65d9c2027"
server_key_version: "encryptionkey@0"
I20260929 11:31:44.618423 10057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:44.636920 10057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:44.641618 10057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:44.641896 10057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:44.642959 10057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:44.644485 10057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:44.644572 10057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:44.644665 10057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:44.644712 10057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:44.677974 10057 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:36951
I20260929 11:31:44.678038 10184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:36951 every 8 connection(s)
I20260929 11:31:44.679386 10057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:31:44.687791 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 10057
I20260929 11:31:44.688154 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:31:44.688634 24164 external_mini_cluster.cc:1428] Setting key 4b3a05e782bce7220871816bc2fedf2e
Sep 29 11:31:44 dist-test-slave-qk05 krb5kdc[9951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681504, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:31:44.702416 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:44.682061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:35271 (local address 127.23.153.62:46565)
0929 11:31:44.682546 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:44.682552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:44.683648 (+ 1096us) server_negotiation.cc:407] Connection header received
0929 11:31:44.685643 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:44.685649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:44.685751 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:44.685853 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:44.687755 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:44.688559 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:44.690546 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:44.691110 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:44.691243 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:44.695819 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:44.695825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:44.695829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:44.695870 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:44.698548 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:44.699716 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:44.699723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:44.699727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:44.699922 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:44.700408 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:44.700413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:44.700416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:44.700791 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:44.700948 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:44.701795 (+ 847us) server_negotiation.cc:299] Negotiation successful
0929 11:31:44.702053 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":239,"threads_started":1}
I20260929 11:31:44.703346 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:44.682673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:46565 (local address 127.23.153.1:35271)
0929 11:31:44.683444 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:44.683491 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:44.685291 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:44.686166 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:44.686176 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:44.686774 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:31:44.687511 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:44.687535 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:44.688756 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:44.688761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:44.690284 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:44.690305 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:44.691419 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:44.691427 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:44.691780 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:44.692576 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:31:44.692609 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:31:44.695469 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:31:44.698774 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:31:44.698786 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:31:44.698807 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:31:44.699479 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:31:44.700045 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:31:44.700059 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:31:44.700061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:31:44.700273 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:31:44.700920 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:31:44.700927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:31:44.701359 (+ 432us) client_negotiation.cc:769] Sending connection context
0929 11:31:44.701633 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 11:31:44.702092 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":326,"threads_started":1}
I20260929 11:31:44.705085 10185 heartbeater.cc:347] Connected to a master server at 127.23.153.62:46565
I20260929 11:31:44.705471 10185 heartbeater.cc:464] Registering TS with master...
I20260929 11:31:44.706458 10185 heartbeater.cc:511] Master 127.23.153.62:46565 requested a full tablet report, sending...
I20260929 11:31:44.709064 9996 ts_manager.cc:194] Registered new tserver with Master: 92445029d03a47289d829b5fc7205929 (127.23.153.1:36951)
I20260929 11:31:44.710683 9996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:35271
WARNING: no policy specified for kudu/127.23.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
I20260929 11:31:44.747114 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:46565
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:31:45.098798 10191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:45.099164 10191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:45.099308 10191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:45.106968 10191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:45.107208 10191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:45.107374 10191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:31:45.117796 10191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:46565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:45.119180 10191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:45.120499 10191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:45.134316 10196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:45.134306 10199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:45.134271 10197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:45.135243 10191 server_base.cc:1034] running on GCE node
I20260929 11:31:45.135900 10191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:45.136946 10191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:45.138193 10191 hybrid_clock.cc:648] HybridClock initialized: now 1790681505138134 us; error 52 us; skew 500 ppm
I20260929 11:31:45.138553 10191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:31:45 dist-test-slave-qk05 krb5kdc[9951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681505, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:31:45.142201 10191 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:31:45.144089 10191 webserver.cc:460] Webserver started at http://127.23.153.2:45203/ using document root <none> and password file <none>
I20260929 11:31:45.144853 10191 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:45.144982 10191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:45.145308 10191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:45.148494 10191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "2d2f40a06324496997124cbd41a603a1"
format_stamp: "Formatted at 2026-09-29 11:31:45 on dist-test-slave-qk05"
server_key: "217c58ee01ca4561d7c1b2170bb8e638"
server_key_iv: "3d0db8f05f56c071bb953a73ae8923d8"
server_key_version: "encryptionkey@0"
I20260929 11:31:45.149339 10191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "2d2f40a06324496997124cbd41a603a1"
format_stamp: "Formatted at 2026-09-29 11:31:45 on dist-test-slave-qk05"
server_key: "217c58ee01ca4561d7c1b2170bb8e638"
server_key_iv: "3d0db8f05f56c071bb953a73ae8923d8"
server_key_version: "encryptionkey@0"
I20260929 11:31:45.155228 10191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 11:31:45.159219 10206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:45.160923 10191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:31:45.161129 10191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "2d2f40a06324496997124cbd41a603a1"
format_stamp: "Formatted at 2026-09-29 11:31:45 on dist-test-slave-qk05"
server_key: "217c58ee01ca4561d7c1b2170bb8e638"
server_key_iv: "3d0db8f05f56c071bb953a73ae8923d8"
server_key_version: "encryptionkey@0"
I20260929 11:31:45.161348 10191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:45.183671 10191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:45.188292 10191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:45.188611 10191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:45.189658 10191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:45.191190 10191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:45.191284 10191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:45.191411 10191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:45.191463 10191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:45.223969 10191 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:35869
I20260929 11:31:45.224005 10318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:35869 every 8 connection(s)
I20260929 11:31:45.225351 10191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:31:45.235477 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 10191
I20260929 11:31:45.235656 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:31:45.236079 24164 external_mini_cluster.cc:1428] Setting key 0b5672c42be06f4bfdeb983d2192cc12
Sep 29 11:31:45 dist-test-slave-qk05 krb5kdc[9951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681505, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:31:45.251305 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:45.228209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:38617 (local address 127.23.153.62:46565)
0929 11:31:45.228675 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:45.228682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:45.229865 (+ 1183us) server_negotiation.cc:407] Connection header received
0929 11:31:45.232026 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:45.232032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:45.232139 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:45.232248 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:45.234858 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.236045 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.238120 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.238817 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.238950 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:45.243892 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:45.243898 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:45.243902 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:45.243944 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:45.246723 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:45.247932 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:45.247939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:45.247942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:45.248131 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:45.248634 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:45.248639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:45.248642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:45.249007 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:45.249141 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:45.250700 (+ 1559us) server_negotiation.cc:299] Negotiation successful
0929 11:31:45.250929 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":335,"thread_start_us":214,"threads_started":1}
I20260929 11:31:45.252830 10322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:45.228841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:46565 (local address 127.23.153.2:38617)
0929 11:31:45.229673 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:45.229723 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:45.231576 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:45.232503 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:45.232514 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:45.233772 (+ 1258us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:31:45.234494 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:45.234517 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.236308 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.236315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:45.237839 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:45.237858 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.239171 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.239183 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:45.239595 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:45.240440 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:31:45.240480 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:31:45.243551 (+ 3071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:31:45.246974 (+ 3423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:31:45.246987 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:31:45.247031 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:31:45.247776 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:31:45.248282 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:31:45.248286 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:31:45.248288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:31:45.248501 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:31:45.249167 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:31:45.249177 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:31:45.249885 (+ 708us) client_negotiation.cc:769] Sending connection context
0929 11:31:45.250582 (+ 697us) client_negotiation.cc:240] Negotiation successful
0929 11:31:45.251307 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":345,"threads_started":1}
I20260929 11:31:45.254503 10319 heartbeater.cc:347] Connected to a master server at 127.23.153.62:46565
I20260929 11:31:45.254833 10319 heartbeater.cc:464] Registering TS with master...
I20260929 11:31:45.255941 10319 heartbeater.cc:511] Master 127.23.153.62:46565 requested a full tablet report, sending...
I20260929 11:31:45.258075 9996 ts_manager.cc:194] Registered new tserver with Master: 2d2f40a06324496997124cbd41a603a1 (127.23.153.2:35869)
I20260929 11:31:45.258953 9996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:38617
WARNING: no policy specified for kudu/127.23.153.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
I20260929 11:31:45.296396 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:46565
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:31:45.681658 10325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:45.681972 10325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:45.682125 10325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:45.690260 10325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:45.690444 10325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:45.690680 10325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:31:45.701921 10325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:46565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:45.703346 10325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:45.704746 10325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 11:31:45.714957 10185 heartbeater.cc:503] Master 127.23.153.62:46565 was elected leader, sending a full tablet report...
W20260929 11:31:45.722040 10330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:45.722132 10333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:45.722076 10331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:45.723177 10325 server_base.cc:1034] running on GCE node
I20260929 11:31:45.723903 10325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:45.724982 10325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:45.726251 10325 hybrid_clock.cc:648] HybridClock initialized: now 1790681505726192 us; error 57 us; skew 500 ppm
I20260929 11:31:45.726663 10325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:31:45 dist-test-slave-qk05 krb5kdc[9951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681505, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:31:45.730822 10325 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:31:45.733000 10325 webserver.cc:460] Webserver started at http://127.23.153.3:40535/ using document root <none> and password file <none>
I20260929 11:31:45.734172 10325 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:45.734512 10325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:45.734941 10325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:45.738322 10325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "3ff49f99881e4cf2982600214410d3fa"
format_stamp: "Formatted at 2026-09-29 11:31:45 on dist-test-slave-qk05"
server_key: "60c63642b51f64fb7b3a6490a2725035"
server_key_iv: "0c663539fa223a673a613a3c4148ce35"
server_key_version: "encryptionkey@0"
I20260929 11:31:45.739296 10325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "3ff49f99881e4cf2982600214410d3fa"
format_stamp: "Formatted at 2026-09-29 11:31:45 on dist-test-slave-qk05"
server_key: "60c63642b51f64fb7b3a6490a2725035"
server_key_iv: "0c663539fa223a673a613a3c4148ce35"
server_key_version: "encryptionkey@0"
I20260929 11:31:45.745414 10325 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:31:45.749588 10340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:45.751356 10325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:31:45.751543 10325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "3ff49f99881e4cf2982600214410d3fa"
format_stamp: "Formatted at 2026-09-29 11:31:45 on dist-test-slave-qk05"
server_key: "60c63642b51f64fb7b3a6490a2725035"
server_key_iv: "0c663539fa223a673a613a3c4148ce35"
server_key_version: "encryptionkey@0"
I20260929 11:31:45.751832 10325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:45.777212 10325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:45.781878 10325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:45.782186 10325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:45.783284 10325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:45.784747 10325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:45.784829 10325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:45.784945 10325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:45.784996 10325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:45.820815 10325 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:33423
I20260929 11:31:45.820889 10452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:33423 every 8 connection(s)
I20260929 11:31:45.822235 10325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:31:45.824275 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 10325
I20260929 11:31:45.824467 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:31:45.824957 24164 external_mini_cluster.cc:1428] Setting key 4aec1c689f354ed151104eba88587a1f
Sep 29 11:31:45 dist-test-slave-qk05 krb5kdc[9951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681505, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:31:45.846884 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:45.825414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:60713 (local address 127.23.153.62:46565)
0929 11:31:45.828322 (+ 2908us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:45.828326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:45.828351 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:31:45.829802 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:45.829808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:45.829934 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:45.830091 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:45.832012 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.832907 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.834895 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.835473 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.835605 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:45.839873 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:45.839881 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:45.839895 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:45.839942 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:45.843406 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:45.844390 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:45.844396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:45.844399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:45.844545 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:45.845078 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:45.845083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:45.845085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:45.845402 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:45.845536 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:45.846349 (+ 813us) server_negotiation.cc:299] Negotiation successful
0929 11:31:45.846566 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2796,"thread_start_us":2659,"threads_started":1}
I20260929 11:31:45.847918 10456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:45.826401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:46565 (local address 127.23.153.3:60713)
0929 11:31:45.827457 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:45.827510 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:45.829375 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:45.830415 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:45.830426 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:45.831061 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:31:45.831750 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:45.831774 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.833127 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.833133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:45.834569 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:45.834658 (+ 89us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.835752 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.835758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:45.836004 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:45.836883 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:31:45.836915 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:31:45.839605 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:31:45.843634 (+ 4029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:31:45.843645 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:31:45.843662 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:31:45.844227 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:31:45.844719 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:31:45.844723 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:31:45.844726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:31:45.844946 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:31:45.845536 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:31:45.845558 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:31:45.845935 (+ 377us) client_negotiation.cc:769] Sending connection context
0929 11:31:45.846205 (+ 270us) client_negotiation.cc:240] Negotiation successful
0929 11:31:45.846671 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":384,"threads_started":1}
I20260929 11:31:45.849562 10453 heartbeater.cc:347] Connected to a master server at 127.23.153.62:46565
I20260929 11:31:45.849900 10453 heartbeater.cc:464] Registering TS with master...
I20260929 11:31:45.850703 10453 heartbeater.cc:511] Master 127.23.153.62:46565 requested a full tablet report, sending...
I20260929 11:31:45.852650 9996 ts_manager.cc:194] Registered new tserver with Master: 3ff49f99881e4cf2982600214410d3fa (127.23.153.3:33423)
I20260929 11:31:45.853521 9996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:60713
I20260929 11:31:45.854012 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:31:45.876273 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:45.858592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47642 (local address 127.23.153.62:46565)
0929 11:31:45.858819 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:45.858826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:45.859206 (+ 380us) server_negotiation.cc:407] Connection header received
0929 11:31:45.859350 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:45.859354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:45.859425 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:45.859516 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:45.861010 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.861783 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.863260 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.863707 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.863812 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:45.867508 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:45.867516 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:45.867521 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:45.867592 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:45.871720 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:45.873225 (+ 1505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:45.873231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:45.873234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:45.873422 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:45.874165 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:45.874170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:45.874173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:45.874887 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:45.875240 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:45.875724 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 11:31:45.875926 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 11:31:45.881110 9996 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:31:45.889709 9996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:31:45.915977 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:45.904982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45742 (local address 127.23.153.1:36951)
0929 11:31:45.905696 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:45.905707 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:45.906847 (+ 1140us) server_negotiation.cc:407] Connection header received
0929 11:31:45.907161 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:45.907171 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:45.907431 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:45.907614 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:45.908859 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.910471 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.913258 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.914789 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.914978 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:45.915234 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:45.915492 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 11:31:45.915676 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":1081,"threads_started":1}
I20260929 11:31:45.916023 10469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:45.904812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:35869 (local address 127.0.0.1:56452)
0929 11:31:45.906255 (+ 1443us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:45.906299 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:45.906486 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:45.907876 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:45.907885 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:45.907940 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:45.909215 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:45.909228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.910929 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.910934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:45.912987 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:45.913003 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.914708 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.914715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:45.914956 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:45.915055 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 11:31:45.915214 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 11:31:45.915450 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"mutex_wait_us":135,"spinlock_wait_cycles":11648,"thread_start_us":1614,"threads_started":1}
I20260929 11:31:45.917001 10468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:45.905171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.23.153.2:35869)
0929 11:31:45.907090 (+ 1919us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:45.907098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:45.907121 (+ 23us) server_negotiation.cc:407] Connection header received
0929 11:31:45.907210 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:45.907215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:45.907422 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:45.907618 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:45.909420 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.910795 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.913191 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.914474 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.914615 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:45.914839 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:45.916496 (+ 1657us) server_negotiation.cc:299] Negotiation successful
0929 11:31:45.916695 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1737,"thread_start_us":1615,"threads_started":1}
I20260929 11:31:45.917088 10466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:45.904813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:36951 (local address 127.0.0.1:45742)
0929 11:31:45.906731 (+ 1918us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:45.906748 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:45.906948 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:45.907876 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:45.907885 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:45.907938 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:45.908620 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:45.908639 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.910614 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.910619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:45.913022 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:45.913039 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.915092 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.915099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:45.915274 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:45.915306 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 11:31:45.915378 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 11:31:45.916712 (+ 1334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1786,"mutex_wait_us":1741,"spinlock_wait_cycles":3712,"thread_start_us":523,"threads_started":1}
I20260929 11:31:45.919008 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:45.907497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:33423 (local address 127.0.0.1:57058)
0929 11:31:45.908836 (+ 1339us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:45.908854 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:45.908957 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:45.911354 (+ 2397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:45.911360 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:45.911390 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:45.911886 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:45.911900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.913993 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.913999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:45.916255 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:45.916273 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.918186 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.918193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:45.918361 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:45.918396 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:31:45.918493 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 11:31:45.918639 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"mutex_wait_us":5,"thread_start_us":2411,"threads_started":1}
I20260929 11:31:45.919816 10471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:45.907916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57058 (local address 127.23.153.3:33423)
0929 11:31:45.910639 (+ 2723us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:45.910649 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:45.910684 (+ 35us) server_negotiation.cc:407] Connection header received
0929 11:31:45.910788 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:45.910796 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:45.911117 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:45.911336 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:45.912105 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.913823 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.916432 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:45.917655 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:45.917789 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:45.918019 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:45.919297 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0929 11:31:45.919492 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2556,"thread_start_us":2374,"threads_started":1}
I20260929 11:31:45.921005 10120 tablet_service.cc:1467] Processing CreateTablet for tablet 109696619229424e89217c1396eb1085 (DEFAULT_TABLE table=test-table [id=fd49317b9f6f4f2795e08a9609e2ff82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:31:45.921001 10254 tablet_service.cc:1467] Processing CreateTablet for tablet 109696619229424e89217c1396eb1085 (DEFAULT_TABLE table=test-table [id=fd49317b9f6f4f2795e08a9609e2ff82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:31:45.922297 10120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109696619229424e89217c1396eb1085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:45.922318 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109696619229424e89217c1396eb1085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:45.927124 10388 tablet_service.cc:1467] Processing CreateTablet for tablet 109696619229424e89217c1396eb1085 (DEFAULT_TABLE table=test-table [id=fd49317b9f6f4f2795e08a9609e2ff82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:31:45.930022 10388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 109696619229424e89217c1396eb1085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:45.934917 10472 tablet_bootstrap.cc:492] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1: Bootstrap starting.
I20260929 11:31:45.938773 10472 tablet_bootstrap.cc:654] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:45.940053 10472 log.cc:824] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:45.941092 10473 tablet_bootstrap.cc:492] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929: Bootstrap starting.
I20260929 11:31:45.945453 10473 tablet_bootstrap.cc:654] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:45.945693 10472 tablet_bootstrap.cc:492] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1: No bootstrap required, opened a new log
I20260929 11:31:45.946103 10472 ts_tablet_manager.cc:1397] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260929 11:31:45.946887 10475 tablet_bootstrap.cc:492] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa: Bootstrap starting.
I20260929 11:31:45.948523 10473 log.cc:824] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:45.951179 10475 tablet_bootstrap.cc:654] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:45.953423 10475 log.cc:824] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:45.953873 10473 tablet_bootstrap.cc:492] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929: No bootstrap required, opened a new log
I20260929 11:31:45.954258 10473 ts_tablet_manager.cc:1397] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260929 11:31:45.958312 10475 tablet_bootstrap.cc:492] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa: No bootstrap required, opened a new log
I20260929 11:31:45.958366 10472 raft_consensus.cc:348] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92445029d03a47289d829b5fc7205929" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 36951 } } peers { permanent_uuid: "2d2f40a06324496997124cbd41a603a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35869 } } peers { permanent_uuid: "3ff49f99881e4cf2982600214410d3fa" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33423 } }
I20260929 11:31:45.958828 10475 ts_tablet_manager.cc:1397] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20260929 11:31:45.958899 10472 raft_consensus.cc:374] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:45.959007 10472 raft_consensus.cc:729] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2f40a06324496997124cbd41a603a1, State: Initialized, Role: FOLLOWER
I20260929 11:31:45.959724 10472 consensus_queue.cc:260] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92445029d03a47289d829b5fc7205929" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 36951 } } peers { permanent_uuid: "2d2f40a06324496997124cbd41a603a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35869 } } peers { permanent_uuid: "3ff49f99881e4cf2982600214410d3fa" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33423 } }
I20260929 11:31:45.961102 10319 heartbeater.cc:503] Master 127.23.153.62:46565 was elected leader, sending a full tablet report...
I20260929 11:31:45.961901 10472 ts_tablet_manager.cc:1428] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20260929 11:31:45.966915 10473 raft_consensus.cc:348] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92445029d03a47289d829b5fc7205929" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 36951 } } peers { permanent_uuid: "2d2f40a06324496997124cbd41a603a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35869 } } peers { permanent_uuid: "3ff49f99881e4cf2982600214410d3fa" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33423 } }
I20260929 11:31:45.967521 10473 raft_consensus.cc:374] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:45.967701 10473 raft_consensus.cc:729] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92445029d03a47289d829b5fc7205929, State: Initialized, Role: FOLLOWER
I20260929 11:31:45.968525 10473 consensus_queue.cc:260] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92445029d03a47289d829b5fc7205929" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 36951 } } peers { permanent_uuid: "2d2f40a06324496997124cbd41a603a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35869 } } peers { permanent_uuid: "3ff49f99881e4cf2982600214410d3fa" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33423 } }
I20260929 11:31:45.970999 10473 ts_tablet_manager.cc:1428] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929: Time spent starting tablet: real 0.016s user 0.018s sys 0.000s
I20260929 11:31:45.970835 10475 raft_consensus.cc:348] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92445029d03a47289d829b5fc7205929" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 36951 } } peers { permanent_uuid: "2d2f40a06324496997124cbd41a603a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35869 } } peers { permanent_uuid: "3ff49f99881e4cf2982600214410d3fa" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33423 } }
I20260929 11:31:45.971443 10475 raft_consensus.cc:374] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:45.971552 10475 raft_consensus.cc:729] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff49f99881e4cf2982600214410d3fa, State: Initialized, Role: FOLLOWER
I20260929 11:31:45.972165 10475 consensus_queue.cc:260] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92445029d03a47289d829b5fc7205929" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 36951 } } peers { permanent_uuid: "2d2f40a06324496997124cbd41a603a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35869 } } peers { permanent_uuid: "3ff49f99881e4cf2982600214410d3fa" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33423 } }
I20260929 11:31:45.973524 10453 heartbeater.cc:503] Master 127.23.153.62:46565 was elected leader, sending a full tablet report...
I20260929 11:31:45.974444 10475 ts_tablet_manager.cc:1428] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260929 11:31:45.977569 10320 tablet.cc:2369] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:31:46.004277 10478 raft_consensus.cc:482] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:31:46.004832 10478 raft_consensus.cc:504] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92445029d03a47289d829b5fc7205929" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 36951 } } peers { permanent_uuid: "2d2f40a06324496997124cbd41a603a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35869 } } peers { permanent_uuid: "3ff49f99881e4cf2982600214410d3fa" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33423 } }
I20260929 11:31:46.007160 10478 leader_election.cc:288] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92445029d03a47289d829b5fc7205929 (127.23.153.1:36951), 3ff49f99881e4cf2982600214410d3fa (127.23.153.3:33423)
I20260929 11:31:46.015782 10471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:46.008194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:53441 (local address 127.23.153.3:33423)
0929 11:31:46.008380 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:46.008385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:46.008854 (+ 469us) server_negotiation.cc:407] Connection header received
0929 11:31:46.009007 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:46.009011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:46.009090 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:46.009185 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:46.009980 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:46.011178 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:46.013906 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:46.014727 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:46.014911 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:46.014994 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:46.015369 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 11:31:46.015511 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 11:31:46.016187 10482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:46.007587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:33423 (local address 127.23.153.2:53441)
0929 11:31:46.008747 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:46.008765 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:46.008885 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:46.009232 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:46.009237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:46.009298 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:46.009752 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:46.009765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:46.011331 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:46.011337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:46.013647 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:46.013657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:46.014986 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:46.014993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:46.015190 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:46.015218 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 11:31:46.015315 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 11:31:46.015469 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":22,"spinlock_wait_cycles":4352,"thread_start_us":822,"threads_started":1}
I20260929 11:31:46.016819 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:46.007407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:36951 (local address 127.23.153.2:53323)
0929 11:31:46.008267 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:46.008288 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:46.008576 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:46.009069 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:46.009074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:46.009096 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:46.009593 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:46.009608 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:46.011105 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:46.011110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:46.013621 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:46.013645 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:46.015887 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:46.015895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:46.016095 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:46.016155 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 11:31:46.016270 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 11:31:46.016425 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":40,"spinlock_wait_cycles":14720,"thread_start_us":541,"threads_started":1}
I20260929 11:31:46.016914 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:46.007738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:53323 (local address 127.23.153.1:36951)
0929 11:31:46.007938 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:46.007945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:46.008409 (+ 464us) server_negotiation.cc:407] Connection header received
0929 11:31:46.008827 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:46.008833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:46.008920 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:46.009028 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:46.009805 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:46.010856 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:46.013874 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:46.015413 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:46.015627 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:46.015708 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:46.016496 (+ 788us) server_negotiation.cc:299] Negotiation successful
0929 11:31:46.016674 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 11:31:46.017382 10408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109696619229424e89217c1396eb1085" candidate_uuid: "2d2f40a06324496997124cbd41a603a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff49f99881e4cf2982600214410d3fa" is_pre_election: true
I20260929 11:31:46.018026 10408 raft_consensus.cc:2454] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d2f40a06324496997124cbd41a603a1 in term 0.
I20260929 11:31:46.017947 10140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109696619229424e89217c1396eb1085" candidate_uuid: "2d2f40a06324496997124cbd41a603a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92445029d03a47289d829b5fc7205929" is_pre_election: true
I20260929 11:31:46.018507 10140 raft_consensus.cc:2454] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d2f40a06324496997124cbd41a603a1 in term 0.
I20260929 11:31:46.018901 10209 leader_election.cc:302] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d2f40a06324496997124cbd41a603a1, 3ff49f99881e4cf2982600214410d3fa; no voters:
I20260929 11:31:46.019428 10478 raft_consensus.cc:2781] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:31:46.019599 10478 raft_consensus.cc:482] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:31:46.019701 10478 raft_consensus.cc:3037] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:46.023236 10478 raft_consensus.cc:504] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92445029d03a47289d829b5fc7205929" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 36951 } } peers { permanent_uuid: "2d2f40a06324496997124cbd41a603a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35869 } } peers { permanent_uuid: "3ff49f99881e4cf2982600214410d3fa" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33423 } }
I20260929 11:31:46.024083 10478 leader_election.cc:288] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [CANDIDATE]: Term 1 election: Requested vote from peers 92445029d03a47289d829b5fc7205929 (127.23.153.1:36951), 3ff49f99881e4cf2982600214410d3fa (127.23.153.3:33423)
I20260929 11:31:46.024937 10408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109696619229424e89217c1396eb1085" candidate_uuid: "2d2f40a06324496997124cbd41a603a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff49f99881e4cf2982600214410d3fa"
I20260929 11:31:46.025024 10140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "109696619229424e89217c1396eb1085" candidate_uuid: "2d2f40a06324496997124cbd41a603a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92445029d03a47289d829b5fc7205929"
I20260929 11:31:46.025194 10408 raft_consensus.cc:3037] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:46.025276 10140 raft_consensus.cc:3037] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:46.028893 10408 raft_consensus.cc:2454] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d2f40a06324496997124cbd41a603a1 in term 1.
I20260929 11:31:46.029208 10140 raft_consensus.cc:2454] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d2f40a06324496997124cbd41a603a1 in term 1.
I20260929 11:31:46.029575 10209 leader_election.cc:302] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d2f40a06324496997124cbd41a603a1, 3ff49f99881e4cf2982600214410d3fa; no voters:
I20260929 11:31:46.030243 10478 raft_consensus.cc:2781] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:31:46.030725 10478 raft_consensus.cc:686] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d2f40a06324496997124cbd41a603a1, State: Running, Role: LEADER
I20260929 11:31:46.031299 10478 consensus_queue.cc:237] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92445029d03a47289d829b5fc7205929" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 36951 } } peers { permanent_uuid: "2d2f40a06324496997124cbd41a603a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35869 } } peers { permanent_uuid: "3ff49f99881e4cf2982600214410d3fa" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33423 } }
I20260929 11:31:46.036370 9996 catalog_manager.cc:5495] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d2f40a06324496997124cbd41a603a1 (127.23.153.2). New cstate: current_term: 1 leader_uuid: "2d2f40a06324496997124cbd41a603a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92445029d03a47289d829b5fc7205929" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 36951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d2f40a06324496997124cbd41a603a1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ff49f99881e4cf2982600214410d3fa" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33423 } health_report { overall_health: UNKNOWN } } }
W20260929 11:31:46.074132 10454 tablet.cc:2369] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:31:46.079681 10468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:46.073451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.23.153.2:35869)
0929 11:31:46.073703 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:46.073709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:46.073789 (+ 80us) server_negotiation.cc:407] Connection header received
0929 11:31:46.074039 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:46.074043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:46.074125 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:46.074262 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:31:46.074679 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:46.075432 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:46.077112 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:46.077555 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:46.077675 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:46.078384 (+ 709us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:31:46.078965 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:31:46.079198 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:46.079370 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 11:31:46.079424 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260929 11:31:46.089960 10140 raft_consensus.cc:1261] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929 [term 1 FOLLOWER]: Refusing update from remote peer 2d2f40a06324496997124cbd41a603a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:31:46.089975 10408 raft_consensus.cc:1261] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa [term 1 FOLLOWER]: Refusing update from remote peer 2d2f40a06324496997124cbd41a603a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:31:46.091161 10483 consensus_queue.cc:1035] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92445029d03a47289d829b5fc7205929" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 36951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:31:46.091562 10478 consensus_queue.cc:1035] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ff49f99881e4cf2982600214410d3fa" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 33423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:31:46.103118 10486 mvcc.cc:205] Tried to move back new op lower bound from 7334631448928325632 to 7334631448708231168. Current Snapshot: MvccSnapshot[applied={T|T < 7334631448928325632}]
I20260929 11:31:46.105448 10488 mvcc.cc:205] Tried to move back new op lower bound from 7334631448928325632 to 7334631448708231168. Current Snapshot: MvccSnapshot[applied={T|T < 7334631448928325632}]
I20260929 11:31:46.123809 9995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:31:46.124169 9995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:31:46.128501 9995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30df268f31cd43b99095af50e58fc1bf: Sending DeleteTablet for 3 replicas of tablet 109696619229424e89217c1396eb1085
I20260929 11:31:46.129936 10120 tablet_service.cc:1514] Processing DeleteTablet for tablet 109696619229424e89217c1396eb1085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:46 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:45742
I20260929 11:31:46.130139 10388 tablet_service.cc:1514] Processing DeleteTablet for tablet 109696619229424e89217c1396eb1085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:46 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:57058
I20260929 11:31:46.130820 10254 tablet_service.cc:1514] Processing DeleteTablet for tablet 109696619229424e89217c1396eb1085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:46 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:56452
I20260929 11:31:46.130996 10497 tablet_replica.cc:330] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa: stopping tablet replica
I20260929 11:31:46.131501 10497 raft_consensus.cc:2229] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:31:46.131539 10496 tablet_replica.cc:330] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929: stopping tablet replica
I20260929 11:31:46.131932 10496 raft_consensus.cc:2229] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:31:46.132063 10497 raft_consensus.cc:2258] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:31:46.132195 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 10057
I20260929 11:31:46.132390 10496 raft_consensus.cc:2258] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:31:46.134289 10497 ts_tablet_manager.cc:1905] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:31:46.136933 10496 ts_tablet_manager.cc:1905] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:31:46.141844 10497 ts_tablet_manager.cc:1918] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:31:46.142079 10497 log.cc:1196] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/109696619229424e89217c1396eb1085
I20260929 11:31:46.142656 10497 ts_tablet_manager.cc:1939] T 109696619229424e89217c1396eb1085 P 3ff49f99881e4cf2982600214410d3fa: Deleting consensus metadata
I20260929 11:31:46.143613 10498 tablet_replica.cc:330] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1: stopping tablet replica
I20260929 11:31:46.144199 10498 raft_consensus.cc:2229] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:31:46.144764 9982 catalog_manager.cc:4841] TS 3ff49f99881e4cf2982600214410d3fa (127.23.153.3:33423): tablet 109696619229424e89217c1396eb1085 (table test-table [id=fd49317b9f6f4f2795e08a9609e2ff82]) successfully deleted
I20260929 11:31:46.147320 10496 ts_tablet_manager.cc:1918] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:31:46.147524 10496 log.cc:1196] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/109696619229424e89217c1396eb1085
I20260929 11:31:46.148113 10496 ts_tablet_manager.cc:1939] T 109696619229424e89217c1396eb1085 P 92445029d03a47289d829b5fc7205929: Deleting consensus metadata
I20260929 11:31:46.150475 9981 catalog_manager.cc:4841] TS 92445029d03a47289d829b5fc7205929 (127.23.153.1:36951): tablet 109696619229424e89217c1396eb1085 (table test-table [id=fd49317b9f6f4f2795e08a9609e2ff82]) successfully deleted
I20260929 11:31:46.150702 10498 raft_consensus.cc:2258] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 29 11:31:46 dist-test-slave-qk05 krb5kdc[9951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:31:46.153369 10498 ts_tablet_manager.cc:1905] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:31:46.161460 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:46.133412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45758 (local address 127.23.153.1:36951)
0929 11:31:46.133591 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:46.133597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:46.142717 (+ 9120us) server_negotiation.cc:407] Connection header received
0929 11:31:46.143097 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:46.143102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:46.143189 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:46.143311 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:46.145586 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:46.146426 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:46.148015 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:46.148529 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:46.148659 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:46.154216 (+ 5557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:46.154243 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:46.154265 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:46.154323 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:46.158296 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:46.159198 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:46.159206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:46.159209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:46.159363 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:46.159757 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:46.159761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:46.159763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:46.160201 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:46.160527 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:46.160833 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 11:31:46.161085 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 11:31:46.161688 10498 ts_tablet_manager.cc:1918] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:31:46.161891 10498 log.cc:1196] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/109696619229424e89217c1396eb1085
I20260929 11:31:46.162221 10180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:46.162489 10498 ts_tablet_manager.cc:1939] T 109696619229424e89217c1396eb1085 P 2d2f40a06324496997124cbd41a603a1: Deleting consensus metadata
I20260929 11:31:46.164582 9980 catalog_manager.cc:4841] TS 2d2f40a06324496997124cbd41a603a1 (127.23.153.2:35869): tablet 109696619229424e89217c1396eb1085 (table test-table [id=fd49317b9f6f4f2795e08a9609e2ff82]) successfully deleted
I20260929 11:31:46.313269 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 10057
I20260929 11:31:46.329324 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 10191
Sep 29 11:31:46 dist-test-slave-qk05 krb5kdc[9951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.2@KRBTEST.COM
I20260929 11:31:46.350229 10468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:46.330171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.23.153.2:35869)
0929 11:31:46.330350 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:46.330355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:46.330393 (+ 38us) server_negotiation.cc:407] Connection header received
0929 11:31:46.330607 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:46.330611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:46.330679 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:46.330769 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:46.332572 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:46.333979 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:46.336922 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:46.337408 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:46.337553 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:46.343429 (+ 5876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:46.343450 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:46.343467 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:46.343520 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:46.346735 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:46.347818 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:46.347826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:46.347832 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:46.348004 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:46.348567 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:46.348574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:46.348578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:46.349032 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:46.349345 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:46.349693 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 11:31:46.349898 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:31:46.350986 10314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:46.472608 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 10191
I20260929 11:31:46.490002 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 10325
Sep 29 11:31:46 dist-test-slave-qk05 krb5kdc[9951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.3@KRBTEST.COM
I20260929 11:31:46.507874 10471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:46.490833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57070 (local address 127.23.153.3:33423)
0929 11:31:46.491046 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:46.491052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:46.491078 (+ 26us) server_negotiation.cc:407] Connection header received
0929 11:31:46.491212 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:46.491216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:46.491295 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:46.491395 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:46.492854 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:46.493633 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:46.495590 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:46.496034 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:46.496162 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:46.501226 (+ 5064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:46.501247 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:46.501264 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:46.501308 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:46.504311 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:46.505163 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:46.505170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:46.505173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:46.505342 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:46.505901 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:46.505905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:46.505908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:46.506319 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:46.506567 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:46.507351 (+ 784us) server_negotiation.cc:299] Negotiation successful
0929 11:31:46.507555 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 11:31:46.508774 10448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:46.600427 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 10325
I20260929 11:31:46.615166 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 9964
I20260929 11:31:46.616366 10026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:46.729498 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 9964
2026-09-29T11:31:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4519 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T11:31:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:31:46Z Disabled control of system clock
I20260929 11:31:46.804347 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:40203
--webserver_interface=127.23.153.62
--webserver_port=0
--builtin_ntp_servers=127.23.153.20:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:40203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 11:31:47.157588 10507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:47.157915 10507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:47.158015 10507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:47.165759 10507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:31:47.165898 10507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:47.165937 10507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:47.166009 10507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:31:47.166071 10507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:31:47.176147 10507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:40203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:40203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:47.177623 10507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:47.179687 10507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:47.192380 10513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:47.192338 10515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:47.192579 10512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:47.193187 10507 server_base.cc:1034] running on GCE node
I20260929 11:31:47.194061 10507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:47.195812 10507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:47.197114 10507 hybrid_clock.cc:648] HybridClock initialized: now 1790681507197052 us; error 53 us; skew 500 ppm
I20260929 11:31:47.197552 10507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:31:47.200513 10507 webserver.cc:460] Webserver started at http://127.23.153.62:35447/ using document root <none> and password file <none>
I20260929 11:31:47.201411 10507 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:47.201527 10507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:47.201879 10507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:47.205332 10507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "452928c2c664419d86a66de03428301b"
format_stamp: "Formatted at 2026-09-29 11:31:47 on dist-test-slave-qk05"
server_key: "55025f95a77fe08ce13361bfaf668d09"
server_key_iv: "619954ae5ee2abf2707a76b2fb5f611d"
server_key_version: "encryptionkey@0"
I20260929 11:31:47.206331 10507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "452928c2c664419d86a66de03428301b"
format_stamp: "Formatted at 2026-09-29 11:31:47 on dist-test-slave-qk05"
server_key: "55025f95a77fe08ce13361bfaf668d09"
server_key_iv: "619954ae5ee2abf2707a76b2fb5f611d"
server_key_version: "encryptionkey@0"
I20260929 11:31:47.212570 10507 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 11:31:47.216948 10521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:47.218672 10507 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 11:31:47.218906 10507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "452928c2c664419d86a66de03428301b"
format_stamp: "Formatted at 2026-09-29 11:31:47 on dist-test-slave-qk05"
server_key: "55025f95a77fe08ce13361bfaf668d09"
server_key_iv: "619954ae5ee2abf2707a76b2fb5f611d"
server_key_version: "encryptionkey@0"
I20260929 11:31:47.219210 10507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:47.263824 10507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:47.264791 10507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:47.265089 10507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:47.285531 10507 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:40203
I20260929 11:31:47.285593 10572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:40203 every 8 connection(s)
I20260929 11:31:47.287088 10507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:31:47.291689 10573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:47.293177 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 10507
I20260929 11:31:47.293380 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:31:47.293938 24164 external_mini_cluster.cc:1428] Setting key 7f2875bf8d55caa6cb194b95854ca723
I20260929 11:31:47.309861 10573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b: Bootstrap starting.
I20260929 11:31:47.311919 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:47.296700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49634 (local address 127.23.153.62:40203)
0929 11:31:47.298132 (+ 1432us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:47.298146 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:47.298204 (+ 58us) server_negotiation.cc:407] Connection header received
0929 11:31:47.300529 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:47.300582 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:47.301360 (+ 778us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:47.301780 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:47.302544 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:47.304066 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:47.306159 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:47.307132 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:47.307322 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:47.308926 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:47.308942 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:47.308968 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:47.309857 (+ 889us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:47.309971 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:47.310004 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:47.310127 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:47.310422 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 11:31:47.310805 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":902,"thread_start_us":508,"threads_started":1}
I20260929 11:31:47.313910 10573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:47.315135 10573 log.cc:824] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:47.318675 10573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b: No bootstrap required, opened a new log
I20260929 11:31:47.327421 10573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452928c2c664419d86a66de03428301b" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40203 } }
I20260929 11:31:47.327853 10573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:47.327946 10573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 452928c2c664419d86a66de03428301b, State: Initialized, Role: FOLLOWER
I20260929 11:31:47.328544 10573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452928c2c664419d86a66de03428301b" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40203 } }
I20260929 11:31:47.328802 10573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:31:47.328967 10573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:31:47.329154 10573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:47.331561 10573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452928c2c664419d86a66de03428301b" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40203 } }
I20260929 11:31:47.332185 10573 leader_election.cc:302] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 452928c2c664419d86a66de03428301b; no voters:
I20260929 11:31:47.332820 10573 leader_election.cc:288] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:31:47.332993 10578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:31:47.333721 10578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b [term 1 LEADER]: Becoming Leader. State: Replica: 452928c2c664419d86a66de03428301b, State: Running, Role: LEADER
I20260929 11:31:47.334316 10578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452928c2c664419d86a66de03428301b" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40203 } }
I20260929 11:31:47.335064 10573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:31:47.337978 10579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "452928c2c664419d86a66de03428301b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452928c2c664419d86a66de03428301b" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40203 } } }
I20260929 11:31:47.338526 10579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b [sys.catalog]: This master's current role is: LEADER
I20260929 11:31:47.341162 10584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:31:47.340950 10580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 452928c2c664419d86a66de03428301b. Latest consensus state: current_term: 1 leader_uuid: "452928c2c664419d86a66de03428301b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452928c2c664419d86a66de03428301b" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40203 } } }
I20260929 11:31:47.341325 10580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b [sys.catalog]: This master's current role is: LEADER
I20260929 11:31:47.348173 10584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:31:47.358928 10584 catalog_manager.cc:1347] Generated new cluster ID: 70770848e85f4d52b5de97f5b6b2e415
I20260929 11:31:47.359102 10584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:31:47.376442 10584 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:31:47.377962 10584 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:31:47.391959 10584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b: Generated new TSK 0
I20260929 11:31:47.392987 10584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:31:47.399780 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40203
--builtin_ntp_servers=127.23.153.20:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 11:31:47.779177 10597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:47.779480 10597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:47.779731 10597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:47.787371 10597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:47.787515 10597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:47.787765 10597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:31:47.797849 10597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:47.799618 10597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:47.801544 10597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:47.815747 10603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:47.815735 10602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:47.816587 10605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:47.816794 10597 server_base.cc:1034] running on GCE node
I20260929 11:31:47.817570 10597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:47.818670 10597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:47.819953 10597 hybrid_clock.cc:648] HybridClock initialized: now 1790681507819865 us; error 80 us; skew 500 ppm
I20260929 11:31:47.820343 10597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:31:47.822939 10597 webserver.cc:460] Webserver started at http://127.23.153.1:33753/ using document root <none> and password file <none>
I20260929 11:31:47.823822 10597 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:47.823935 10597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:47.824268 10597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:47.827472 10597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "32a6617dbc794f9e8c2dac4cf223b76d"
format_stamp: "Formatted at 2026-09-29 11:31:47 on dist-test-slave-qk05"
server_key: "223d97e07dc5c7d959646b3c2a6de2dd"
server_key_iv: "f8f1dd71111777b2f1acbf1087faa32c"
server_key_version: "encryptionkey@0"
I20260929 11:31:47.828310 10597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "32a6617dbc794f9e8c2dac4cf223b76d"
format_stamp: "Formatted at 2026-09-29 11:31:47 on dist-test-slave-qk05"
server_key: "223d97e07dc5c7d959646b3c2a6de2dd"
server_key_iv: "f8f1dd71111777b2f1acbf1087faa32c"
server_key_version: "encryptionkey@0"
I20260929 11:31:47.834499 10597 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 11:31:47.838564 10611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:47.840648 10597 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:31:47.840948 10597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "32a6617dbc794f9e8c2dac4cf223b76d"
format_stamp: "Formatted at 2026-09-29 11:31:47 on dist-test-slave-qk05"
server_key: "223d97e07dc5c7d959646b3c2a6de2dd"
server_key_iv: "f8f1dd71111777b2f1acbf1087faa32c"
server_key_version: "encryptionkey@0"
I20260929 11:31:47.841342 10597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:47.866585 10597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:47.867575 10597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:47.867904 10597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:47.868964 10597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:47.871161 10597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:47.871289 10597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:47.871410 10597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:47.871483 10597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:47.908938 10597 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:37627
I20260929 11:31:47.909081 10723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:37627 every 8 connection(s)
I20260929 11:31:47.910509 10597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:31:47.919952 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 10597
I20260929 11:31:47.920161 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:31:47.920804 24164 external_mini_cluster.cc:1428] Setting key 0817bdca57efedf3734e41160047c8f7
I20260929 11:31:47.924731 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40203
--builtin_ntp_servers=127.23.153.20:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 11:31:47.925726 10726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:47.913382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:42825 (local address 127.23.153.62:40203)
0929 11:31:47.913902 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:47.913909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:47.915207 (+ 1298us) server_negotiation.cc:407] Connection header received
0929 11:31:47.917250 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:47.917254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:47.917334 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:47.917450 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:47.918806 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:47.920174 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:47.922276 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:47.922815 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:47.922953 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:47.924748 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:47.924755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:47.924758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:47.924779 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:47.924827 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:47.924841 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:47.924922 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:47.925206 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 11:31:47.925326 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":251,"threads_started":1}
I20260929 11:31:47.926834 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:47.914004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:40203 (local address 127.23.153.1:42825)
0929 11:31:47.914989 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:47.915068 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:47.916927 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:47.917718 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:47.917729 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:47.917845 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:31:47.918470 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:47.918488 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:47.920367 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:47.920372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:47.922108 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:47.922126 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:47.923103 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:47.923113 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:47.923424 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:47.924244 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:31:47.924249 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:31:47.924315 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:31:47.924481 (+ 166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:31:47.924949 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:31:47.924962 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:31:47.924972 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 11:31:47.925180 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 11:31:47.925656 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":286,"threads_started":1}
I20260929 11:31:47.928617 10724 heartbeater.cc:347] Connected to a master server at 127.23.153.62:40203
I20260929 11:31:47.928949 10724 heartbeater.cc:464] Registering TS with master...
I20260929 11:31:47.929765 10724 heartbeater.cc:511] Master 127.23.153.62:40203 requested a full tablet report, sending...
I20260929 11:31:47.932384 10538 ts_manager.cc:194] Registered new tserver with Master: 32a6617dbc794f9e8c2dac4cf223b76d (127.23.153.1:37627)
I20260929 11:31:47.934180 10538 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.153.1:42825
W20260929 11:31:48.297130 10728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:48.297458 10728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:48.297586 10728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:48.305567 10728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:48.305712 10728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:48.305860 10728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:31:48.317066 10728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:48.319147 10728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:48.321272 10728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:48.336073 10734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:48.335995 10733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:48.336050 10736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:48.337176 10728 server_base.cc:1034] running on GCE node
I20260929 11:31:48.337914 10728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:48.339165 10728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:48.340449 10728 hybrid_clock.cc:648] HybridClock initialized: now 1790681508340380 us; error 72 us; skew 500 ppm
I20260929 11:31:48.340886 10728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:31:48.343722 10728 webserver.cc:460] Webserver started at http://127.23.153.2:35797/ using document root <none> and password file <none>
I20260929 11:31:48.344549 10728 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:48.344656 10728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:48.344959 10728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:48.348196 10728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "e64630a2cbf84ef38a61319a0eb1aa0b"
format_stamp: "Formatted at 2026-09-29 11:31:48 on dist-test-slave-qk05"
server_key: "c76b0058dd386ac2ec3fe03f401e13a5"
server_key_iv: "da244450096f7c93f206d3d9a1325660"
server_key_version: "encryptionkey@0"
I20260929 11:31:48.349080 10728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "e64630a2cbf84ef38a61319a0eb1aa0b"
format_stamp: "Formatted at 2026-09-29 11:31:48 on dist-test-slave-qk05"
server_key: "c76b0058dd386ac2ec3fe03f401e13a5"
server_key_iv: "da244450096f7c93f206d3d9a1325660"
server_key_version: "encryptionkey@0"
I20260929 11:31:48.355090 10728 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 11:31:48.359138 10742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:48.360883 10728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:31:48.361081 10728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "e64630a2cbf84ef38a61319a0eb1aa0b"
format_stamp: "Formatted at 2026-09-29 11:31:48 on dist-test-slave-qk05"
server_key: "c76b0058dd386ac2ec3fe03f401e13a5"
server_key_iv: "da244450096f7c93f206d3d9a1325660"
server_key_version: "encryptionkey@0"
I20260929 11:31:48.361316 10728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:48.407847 10728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:48.408973 10728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:48.409320 10728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:48.411305 10728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:48.413177 10728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:48.413276 10728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:48.413399 10728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:48.413465 10728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:48.448619 10728 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:35073
I20260929 11:31:48.448673 10854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:35073 every 8 connection(s)
I20260929 11:31:48.450070 10728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:31:48.454841 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 10728
I20260929 11:31:48.455008 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:31:48.455472 24164 external_mini_cluster.cc:1428] Setting key ed412a72f71240e8c615ca156a34398f
I20260929 11:31:48.458395 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40203
--builtin_ntp_servers=127.23.153.20:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 11:31:48.464613 10857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:48.452845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:51423 (local address 127.23.153.62:40203)
0929 11:31:48.453287 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:48.453291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:48.454754 (+ 1463us) server_negotiation.cc:407] Connection header received
0929 11:31:48.457250 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:48.457255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:48.457345 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:48.457441 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:48.458773 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:48.459725 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:48.461578 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:48.462027 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:48.462146 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:48.463727 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:48.463733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:48.463736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:48.463758 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:48.463799 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:48.463812 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:48.463893 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:48.464130 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 11:31:48.464250 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":233,"threads_started":1}
I20260929 11:31:48.465957 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:48.453360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:40203 (local address 127.23.153.2:51423)
0929 11:31:48.454352 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:48.454400 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:48.456905 (+ 2505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:48.457764 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:48.457774 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:48.457891 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:31:48.458545 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:48.458573 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:48.459920 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:48.459924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:48.461359 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:48.461378 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:48.462295 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:48.462304 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:48.462556 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:48.463327 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:31:48.463329 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:31:48.463382 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:31:48.463478 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:31:48.463911 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:31:48.463922 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:31:48.463931 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 11:31:48.464116 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 11:31:48.464600 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":299,"threads_started":1}
I20260929 11:31:48.467773 10855 heartbeater.cc:347] Connected to a master server at 127.23.153.62:40203
I20260929 11:31:48.468123 10855 heartbeater.cc:464] Registering TS with master...
I20260929 11:31:48.469027 10855 heartbeater.cc:511] Master 127.23.153.62:40203 requested a full tablet report, sending...
I20260929 11:31:48.471247 10538 ts_manager.cc:194] Registered new tserver with Master: e64630a2cbf84ef38a61319a0eb1aa0b (127.23.153.2:35073)
I20260929 11:31:48.472363 10538 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.153.2:51423
W20260929 11:31:48.823884 10859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:48.824261 10859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:48.824384 10859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:48.832770 10859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:48.832911 10859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:48.833026 10859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:31:48.843209 10859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:48.844581 10859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:48.846966 10859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:48.861559 10864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:48.861474 10865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:48.861512 10867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:48.862773 10859 server_base.cc:1034] running on GCE node
I20260929 11:31:48.863548 10859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:48.864786 10859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:48.866039 10859 hybrid_clock.cc:648] HybridClock initialized: now 1790681508865997 us; error 40 us; skew 500 ppm
I20260929 11:31:48.866466 10859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:31:48.869496 10859 webserver.cc:460] Webserver started at http://127.23.153.3:46259/ using document root <none> and password file <none>
I20260929 11:31:48.870275 10859 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:48.870389 10859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:48.870723 10859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:48.874019 10859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "7203a603d1d848738cec170c5d7a9433"
format_stamp: "Formatted at 2026-09-29 11:31:48 on dist-test-slave-qk05"
server_key: "ae038196a82c718e9b1f76888cfe6980"
server_key_iv: "8e1d07f4cec8e3fc5e375d6d3ff2579d"
server_key_version: "encryptionkey@0"
I20260929 11:31:48.874912 10859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "7203a603d1d848738cec170c5d7a9433"
format_stamp: "Formatted at 2026-09-29 11:31:48 on dist-test-slave-qk05"
server_key: "ae038196a82c718e9b1f76888cfe6980"
server_key_iv: "8e1d07f4cec8e3fc5e375d6d3ff2579d"
server_key_version: "encryptionkey@0"
I20260929 11:31:48.880887 10859 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 11:31:48.884985 10873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:48.886667 10859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:31:48.886873 10859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "7203a603d1d848738cec170c5d7a9433"
format_stamp: "Formatted at 2026-09-29 11:31:48 on dist-test-slave-qk05"
server_key: "ae038196a82c718e9b1f76888cfe6980"
server_key_iv: "8e1d07f4cec8e3fc5e375d6d3ff2579d"
server_key_version: "encryptionkey@0"
I20260929 11:31:48.887159 10859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:48.911767 10859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:48.912731 10859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:48.913117 10859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:48.914326 10859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:48.916265 10859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:48.916366 10859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:48.916455 10859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:48.916527 10859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:48.938012 10724 heartbeater.cc:503] Master 127.23.153.62:40203 was elected leader, sending a full tablet report...
I20260929 11:31:48.952009 10859 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:44617
I20260929 11:31:48.952171 10985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:44617 every 8 connection(s)
I20260929 11:31:48.953414 10859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:31:48.957029 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 10859
I20260929 11:31:48.957223 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:31:48.957810 24164 external_mini_cluster.cc:1428] Setting key 8429abbc82065ba4b1355ca2a6d443aa
I20260929 11:31:48.968777 10857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:48.956352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:37529 (local address 127.23.153.62:40203)
0929 11:31:48.956609 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:48.956614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:48.958872 (+ 2258us) server_negotiation.cc:407] Connection header received
0929 11:31:48.961102 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:48.961107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:48.961198 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:48.961282 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:48.963122 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:48.963864 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:48.965710 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:48.966170 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:48.966293 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:48.967897 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:48.967903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:48.967906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:48.967928 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:48.967964 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:48.967978 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:48.968082 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:48.968319 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 11:31:48.968437 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 11:31:48.969933 10988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:48.957007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:40203 (local address 127.23.153.3:37529)
0929 11:31:48.958696 (+ 1689us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:48.958743 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:48.960724 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:48.961734 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:48.961749 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:48.961903 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:31:48.962852 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:48.962880 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:48.964043 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:48.964048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:48.965442 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:48.965461 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:48.966460 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:48.966468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:48.966734 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:48.967507 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:31:48.967510 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:31:48.967561 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:31:48.967667 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:31:48.968100 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:31:48.968111 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:31:48.968120 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 11:31:48.968297 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 11:31:48.968786 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":462,"threads_started":1}
I20260929 11:31:48.971426 10986 heartbeater.cc:347] Connected to a master server at 127.23.153.62:40203
I20260929 11:31:48.971800 10986 heartbeater.cc:464] Registering TS with master...
I20260929 11:31:48.972767 10986 heartbeater.cc:511] Master 127.23.153.62:40203 requested a full tablet report, sending...
I20260929 11:31:48.974326 10538 ts_manager.cc:194] Registered new tserver with Master: 7203a603d1d848738cec170c5d7a9433 (127.23.153.3:44617)
I20260929 11:31:48.975430 10538 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.153.3:37529
I20260929 11:31:48.986865 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:31:48.998728 10857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:48.991951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.23.153.62:40203)
0929 11:31:48.992137 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:48.992143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:48.992578 (+ 435us) server_negotiation.cc:407] Connection header received
0929 11:31:48.992763 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:48.992768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:48.992845 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:48.992949 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:48.993467 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:48.994238 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:48.996298 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:48.996762 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:48.996875 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:48.998031 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:48.998037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:48.998040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:48.998073 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:48.998108 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:48.998121 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:48.998241 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:48.998375 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 11:31:48.998439 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 11:31:49.003490 10538 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:31:49.011178 10538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:31:49.048333 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:49.027628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.23.153.2:35073)
0929 11:31:49.038680 (+ 11052us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:49.038690 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:49.038724 (+ 34us) server_negotiation.cc:407] Connection header received
0929 11:31:49.038823 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:49.038830 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:49.039132 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:49.039375 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:49.040491 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.042174 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.045088 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.047058 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.047216 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:49.047509 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:49.047801 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 11:31:49.047997 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10891,"thread_start_us":10752,"threads_started":1}
I20260929 11:31:49.048341 10998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:49.027083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:35073 (local address 127.0.0.1:54654)
0929 11:31:49.037809 (+ 10726us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:49.037907 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:49.038219 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:49.039736 (+ 1517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:49.039744 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:49.039783 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:49.040250 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:49.040267 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.042420 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.042426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:49.044817 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:49.044835 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.047309 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.047317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:49.047548 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:49.047602 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 11:31:49.047728 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 11:31:49.047946 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10292,"thread_start_us":10060,"threads_started":1}
I20260929 11:31:49.051695 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:49.027105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:37627 (local address 127.0.0.1:51710)
0929 11:31:49.038848 (+ 11743us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:49.038870 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:49.038994 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:49.040532 (+ 1538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:49.040537 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:49.040565 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:49.040973 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:49.040987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.044020 (+ 3033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.044026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:49.046998 (+ 2972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:49.047038 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.050741 (+ 3703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.050755 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:49.050986 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:49.051053 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 11:31:49.051165 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 11:31:49.051329 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11593,"thread_start_us":11935,"threads_started":1}
I20260929 11:31:49.052284 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:49.027364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51710 (local address 127.23.153.1:37627)
0929 11:31:49.039624 (+ 12260us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:49.039634 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:49.039668 (+ 34us) server_negotiation.cc:407] Connection header received
0929 11:31:49.040037 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:49.040048 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:49.040327 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:49.040564 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:49.041934 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.043796 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.047316 (+ 3520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.049916 (+ 2600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.050074 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:49.050390 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:49.051553 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0929 11:31:49.051889 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12045,"thread_start_us":12300,"threads_started":1}
I20260929 11:31:49.053485 10790 tablet_service.cc:1467] Processing CreateTablet for tablet 0b0a261833424a748d3a211f8cbd76e2 (DEFAULT_TABLE table=test-table [id=b4d58355ba6d4b758832de85b3d6504c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:31:49.055397 10790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b0a261833424a748d3a211f8cbd76e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:49.057019 10659 tablet_service.cc:1467] Processing CreateTablet for tablet 0b0a261833424a748d3a211f8cbd76e2 (DEFAULT_TABLE table=test-table [id=b4d58355ba6d4b758832de85b3d6504c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:31:49.058821 10659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b0a261833424a748d3a211f8cbd76e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:49.059283 11002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:49.039507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:44617 (local address 127.0.0.1:54266)
0929 11:31:49.041186 (+ 1679us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:49.041205 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:49.041548 (+ 343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:49.046126 (+ 4578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:49.046132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:49.046173 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:49.048373 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:49.048392 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.052593 (+ 4201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.052599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:49.054303 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:49.054323 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.058522 (+ 4199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.058528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:49.058682 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:49.058712 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 11:31:49.058792 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 11:31:49.058913 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1543,"thread_start_us":1607,"threads_started":1}
I20260929 11:31:49.059849 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:49.043476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54266 (local address 127.23.153.3:44617)
0929 11:31:49.044385 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:49.044394 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:49.044461 (+ 67us) server_negotiation.cc:407] Connection header received
0929 11:31:49.044582 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:49.044591 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:49.044913 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:49.045516 (+ 603us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:49.048870 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.052365 (+ 3495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.054647 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.056624 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.056814 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:49.059137 (+ 2323us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:49.059251 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 11:31:49.059480 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":475,"threads_started":1}
I20260929 11:31:49.063791 10921 tablet_service.cc:1467] Processing CreateTablet for tablet 0b0a261833424a748d3a211f8cbd76e2 (DEFAULT_TABLE table=test-table [id=b4d58355ba6d4b758832de85b3d6504c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:31:49.065343 10921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b0a261833424a748d3a211f8cbd76e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:49.073128 11004 tablet_bootstrap.cc:492] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b: Bootstrap starting.
I20260929 11:31:49.075963 11005 tablet_bootstrap.cc:492] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d: Bootstrap starting.
I20260929 11:31:49.077554 11004 tablet_bootstrap.cc:654] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:49.079109 11004 log.cc:824] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:49.080288 11005 tablet_bootstrap.cc:654] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:49.081501 11005 log.cc:824] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:49.082898 11004 tablet_bootstrap.cc:492] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b: No bootstrap required, opened a new log
I20260929 11:31:49.083290 11004 ts_tablet_manager.cc:1397] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260929 11:31:49.084579 11005 tablet_bootstrap.cc:492] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d: No bootstrap required, opened a new log
I20260929 11:31:49.084854 11005 ts_tablet_manager.cc:1397] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 11:31:49.087738 11008 tablet_bootstrap.cc:492] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433: Bootstrap starting.
I20260929 11:31:49.092240 11008 tablet_bootstrap.cc:654] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:49.093300 11005 raft_consensus.cc:348] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a6617dbc794f9e8c2dac4cf223b76d" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 37627 } } peers { permanent_uuid: "e64630a2cbf84ef38a61319a0eb1aa0b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35073 } } peers { permanent_uuid: "7203a603d1d848738cec170c5d7a9433" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44617 } }
I20260929 11:31:49.093304 11004 raft_consensus.cc:348] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a6617dbc794f9e8c2dac4cf223b76d" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 37627 } } peers { permanent_uuid: "e64630a2cbf84ef38a61319a0eb1aa0b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35073 } } peers { permanent_uuid: "7203a603d1d848738cec170c5d7a9433" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44617 } }
I20260929 11:31:49.093719 11005 raft_consensus.cc:374] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:49.093824 11004 raft_consensus.cc:374] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:49.093840 11005 raft_consensus.cc:729] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a6617dbc794f9e8c2dac4cf223b76d, State: Initialized, Role: FOLLOWER
I20260929 11:31:49.093817 11008 log.cc:824] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:49.093930 11004 raft_consensus.cc:729] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e64630a2cbf84ef38a61319a0eb1aa0b, State: Initialized, Role: FOLLOWER
I20260929 11:31:49.094518 11005 consensus_queue.cc:260] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a6617dbc794f9e8c2dac4cf223b76d" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 37627 } } peers { permanent_uuid: "e64630a2cbf84ef38a61319a0eb1aa0b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35073 } } peers { permanent_uuid: "7203a603d1d848738cec170c5d7a9433" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44617 } }
I20260929 11:31:49.094758 11004 consensus_queue.cc:260] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a6617dbc794f9e8c2dac4cf223b76d" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 37627 } } peers { permanent_uuid: "e64630a2cbf84ef38a61319a0eb1aa0b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35073 } } peers { permanent_uuid: "7203a603d1d848738cec170c5d7a9433" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44617 } }
I20260929 11:31:49.097021 11005 ts_tablet_manager.cc:1428] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260929 11:31:49.097386 11004 ts_tablet_manager.cc:1428] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b: Time spent starting tablet: real 0.014s user 0.005s sys 0.008s
I20260929 11:31:49.097673 10855 heartbeater.cc:503] Master 127.23.153.62:40203 was elected leader, sending a full tablet report...
I20260929 11:31:49.098253 11008 tablet_bootstrap.cc:492] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433: No bootstrap required, opened a new log
I20260929 11:31:49.098683 11008 ts_tablet_manager.cc:1397] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260929 11:31:49.104604 11010 raft_consensus.cc:482] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:31:49.104933 11010 raft_consensus.cc:504] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a6617dbc794f9e8c2dac4cf223b76d" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 37627 } } peers { permanent_uuid: "e64630a2cbf84ef38a61319a0eb1aa0b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35073 } } peers { permanent_uuid: "7203a603d1d848738cec170c5d7a9433" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44617 } }
I20260929 11:31:49.106856 11010 leader_election.cc:288] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e64630a2cbf84ef38a61319a0eb1aa0b (127.23.153.2:35073), 7203a603d1d848738cec170c5d7a9433 (127.23.153.3:44617)
I20260929 11:31:49.109511 11008 raft_consensus.cc:348] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a6617dbc794f9e8c2dac4cf223b76d" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 37627 } } peers { permanent_uuid: "e64630a2cbf84ef38a61319a0eb1aa0b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35073 } } peers { permanent_uuid: "7203a603d1d848738cec170c5d7a9433" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44617 } }
I20260929 11:31:49.110033 11008 raft_consensus.cc:374] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:49.110143 11008 raft_consensus.cc:729] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7203a603d1d848738cec170c5d7a9433, State: Initialized, Role: FOLLOWER
I20260929 11:31:49.110879 11008 consensus_queue.cc:260] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a6617dbc794f9e8c2dac4cf223b76d" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 37627 } } peers { permanent_uuid: "e64630a2cbf84ef38a61319a0eb1aa0b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35073 } } peers { permanent_uuid: "7203a603d1d848738cec170c5d7a9433" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44617 } }
I20260929 11:31:49.112195 10986 heartbeater.cc:503] Master 127.23.153.62:40203 was elected leader, sending a full tablet report...
I20260929 11:31:49.112962 11008 ts_tablet_manager.cc:1428] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260929 11:31:49.116065 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:49.107778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:41879 (local address 127.23.153.2:35073)
0929 11:31:49.107992 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:49.107998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:49.108697 (+ 699us) server_negotiation.cc:407] Connection header received
0929 11:31:49.109033 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:49.109039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:49.109139 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:49.109254 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:49.109917 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.110930 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.114213 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.115058 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.115245 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:49.115322 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:49.115651 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 11:31:49.115791 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 11:31:49.116102 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:49.107263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:35073 (local address 127.23.153.1:41879)
0929 11:31:49.108558 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:49.108582 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:49.108840 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:49.109277 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:49.109282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:49.109309 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:49.109732 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:49.109745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.111144 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.111152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:49.113960 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:49.113987 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.115276 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.115284 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:49.115486 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:49.115522 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 11:31:49.115615 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 11:31:49.115767 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1131,"threads_started":1}
I20260929 11:31:49.120337 10810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b0a261833424a748d3a211f8cbd76e2" candidate_uuid: "32a6617dbc794f9e8c2dac4cf223b76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e64630a2cbf84ef38a61319a0eb1aa0b" is_pre_election: true
I20260929 11:31:49.120906 10810 raft_consensus.cc:2454] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32a6617dbc794f9e8c2dac4cf223b76d in term 0.
I20260929 11:31:49.121783 10614 leader_election.cc:302] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32a6617dbc794f9e8c2dac4cf223b76d, e64630a2cbf84ef38a61319a0eb1aa0b; no voters:
I20260929 11:31:49.122090 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:49.107286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:40153 (local address 127.23.153.3:44617)
0929 11:31:49.107520 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:49.107526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:49.116444 (+ 8918us) server_negotiation.cc:407] Connection header received
0929 11:31:49.116729 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:49.116736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:49.116824 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:49.116934 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:49.117339 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.118190 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.120144 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.121033 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.121183 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:49.121240 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:49.121710 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 11:31:49.121845 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 11:31:49.122112 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:49.107550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:44617 (local address 127.23.153.1:40153)
0929 11:31:49.116347 (+ 8797us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:49.116368 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:49.116564 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:49.116981 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:49.116987 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:49.117014 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:49.117194 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:49.117206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.118369 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.118375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:49.119956 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:49.119981 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.121353 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.121361 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:49.121549 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:49.121576 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 11:31:49.121661 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 11:31:49.121816 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8651,"thread_start_us":608,"threads_started":1}
I20260929 11:31:49.122249 11010 raft_consensus.cc:2781] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:31:49.122406 11010 raft_consensus.cc:482] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:31:49.122517 11010 raft_consensus.cc:3037] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:49.123116 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b0a261833424a748d3a211f8cbd76e2" candidate_uuid: "32a6617dbc794f9e8c2dac4cf223b76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7203a603d1d848738cec170c5d7a9433" is_pre_election: true
I20260929 11:31:49.123661 10941 raft_consensus.cc:2454] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32a6617dbc794f9e8c2dac4cf223b76d in term 0.
I20260929 11:31:49.125036 11010 raft_consensus.cc:504] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a6617dbc794f9e8c2dac4cf223b76d" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 37627 } } peers { permanent_uuid: "e64630a2cbf84ef38a61319a0eb1aa0b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35073 } } peers { permanent_uuid: "7203a603d1d848738cec170c5d7a9433" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44617 } }
I20260929 11:31:49.125834 11010 leader_election.cc:288] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [CANDIDATE]: Term 1 election: Requested vote from peers e64630a2cbf84ef38a61319a0eb1aa0b (127.23.153.2:35073), 7203a603d1d848738cec170c5d7a9433 (127.23.153.3:44617)
I20260929 11:31:49.126338 10810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b0a261833424a748d3a211f8cbd76e2" candidate_uuid: "32a6617dbc794f9e8c2dac4cf223b76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e64630a2cbf84ef38a61319a0eb1aa0b"
I20260929 11:31:49.126376 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b0a261833424a748d3a211f8cbd76e2" candidate_uuid: "32a6617dbc794f9e8c2dac4cf223b76d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7203a603d1d848738cec170c5d7a9433"
I20260929 11:31:49.126573 10941 raft_consensus.cc:3037] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:49.126636 10810 raft_consensus.cc:3037] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:49.129051 10941 raft_consensus.cc:2454] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32a6617dbc794f9e8c2dac4cf223b76d in term 1.
I20260929 11:31:49.129168 10810 raft_consensus.cc:2454] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32a6617dbc794f9e8c2dac4cf223b76d in term 1.
I20260929 11:31:49.129621 10613 leader_election.cc:302] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32a6617dbc794f9e8c2dac4cf223b76d, 7203a603d1d848738cec170c5d7a9433; no voters:
I20260929 11:31:49.130018 11010 raft_consensus.cc:2781] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:31:49.130540 11010 raft_consensus.cc:686] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [term 1 LEADER]: Becoming Leader. State: Replica: 32a6617dbc794f9e8c2dac4cf223b76d, State: Running, Role: LEADER
I20260929 11:31:49.131103 11010 consensus_queue.cc:237] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a6617dbc794f9e8c2dac4cf223b76d" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 37627 } } peers { permanent_uuid: "e64630a2cbf84ef38a61319a0eb1aa0b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35073 } } peers { permanent_uuid: "7203a603d1d848738cec170c5d7a9433" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44617 } }
I20260929 11:31:49.136741 10536 catalog_manager.cc:5495] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32a6617dbc794f9e8c2dac4cf223b76d (127.23.153.1). New cstate: current_term: 1 leader_uuid: "32a6617dbc794f9e8c2dac4cf223b76d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a6617dbc794f9e8c2dac4cf223b76d" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 37627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e64630a2cbf84ef38a61319a0eb1aa0b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7203a603d1d848738cec170c5d7a9433" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44617 } health_report { overall_health: UNKNOWN } } }
W20260929 11:31:49.163342 10725 tablet.cc:2369] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:31:49.164769 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:49.158600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51726 (local address 127.23.153.1:37627)
0929 11:31:49.158836 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:49.158842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:49.158866 (+ 24us) server_negotiation.cc:407] Connection header received
0929 11:31:49.159005 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:49.159036 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:49.159155 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:49.159298 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:31:49.159693 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.160425 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.162056 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.162533 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.162670 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:49.163436 (+ 766us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:31:49.164088 (+ 652us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:31:49.164242 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:49.164411 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 11:31:49.164463 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 11:31:49.174746 10941 raft_consensus.cc:1261] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433 [term 1 FOLLOWER]: Refusing update from remote peer 32a6617dbc794f9e8c2dac4cf223b76d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:31:49.174816 10810 raft_consensus.cc:1261] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b [term 1 FOLLOWER]: Refusing update from remote peer 32a6617dbc794f9e8c2dac4cf223b76d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:31:49.175962 11015 consensus_queue.cc:1035] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e64630a2cbf84ef38a61319a0eb1aa0b" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 35073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:31:49.176383 11010 consensus_queue.cc:1035] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7203a603d1d848738cec170c5d7a9433" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 44617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:31:49.189572 11018 mvcc.cc:205] Tried to move back new op lower bound from 7334631461567610880 to 7334631461405245440. Current Snapshot: MvccSnapshot[applied={T|T < 7334631461567610880}]
I20260929 11:31:49.205233 10538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:31:49.205605 10538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:31:49.218768 10538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 452928c2c664419d86a66de03428301b: Sending DeleteTablet for 3 replicas of tablet 0b0a261833424a748d3a211f8cbd76e2
I20260929 11:31:49.220206 10790 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b0a261833424a748d3a211f8cbd76e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:49 UTC) from {username='slave'} at 127.0.0.1:54654
I20260929 11:31:49.220494 10921 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b0a261833424a748d3a211f8cbd76e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:49 UTC) from {username='slave'} at 127.0.0.1:54266
I20260929 11:31:49.221825 10659 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b0a261833424a748d3a211f8cbd76e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:49 UTC) from {username='slave'} at 127.0.0.1:51710
I20260929 11:31:49.222146 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 10597
I20260929 11:31:49.225389 11030 tablet_replica.cc:330] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d: stopping tablet replica
I20260929 11:31:49.225976 11030 raft_consensus.cc:2229] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:31:49.226845 11030 raft_consensus.cc:2258] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:31:49.227819 11029 tablet_replica.cc:330] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433: stopping tablet replica
I20260929 11:31:49.228312 11029 raft_consensus.cc:2229] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:31:49.228848 11029 raft_consensus.cc:2258] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:31:49.231297 11029 ts_tablet_manager.cc:1905] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:31:49.231479 11030 ts_tablet_manager.cc:1905] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:31:49.235528 11028 tablet_replica.cc:330] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b: stopping tablet replica
I20260929 11:31:49.236300 11028 raft_consensus.cc:2229] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:31:49.236900 11028 raft_consensus.cc:2258] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:31:49.239573 11028 ts_tablet_manager.cc:1905] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:31:49.240985 11029 ts_tablet_manager.cc:1918] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:31:49.241204 11029 log.cc:1196] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/0b0a261833424a748d3a211f8cbd76e2
I20260929 11:31:49.241761 11029 ts_tablet_manager.cc:1939] T 0b0a261833424a748d3a211f8cbd76e2 P 7203a603d1d848738cec170c5d7a9433: Deleting consensus metadata
I20260929 11:31:49.242079 11030 ts_tablet_manager.cc:1918] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:31:49.242275 11030 log.cc:1196] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/0b0a261833424a748d3a211f8cbd76e2
I20260929 11:31:49.243542 11030 ts_tablet_manager.cc:1939] T 0b0a261833424a748d3a211f8cbd76e2 P 32a6617dbc794f9e8c2dac4cf223b76d: Deleting consensus metadata
I20260929 11:31:49.243840 10523 catalog_manager.cc:4841] TS 7203a603d1d848738cec170c5d7a9433 (127.23.153.3:44617): tablet 0b0a261833424a748d3a211f8cbd76e2 (table test-table [id=b4d58355ba6d4b758832de85b3d6504c]) successfully deleted
I20260929 11:31:49.245674 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:49.223359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.23.153.1:37627)
0929 11:31:49.223555 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:49.223562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:49.224662 (+ 1100us) server_negotiation.cc:407] Connection header received
0929 11:31:49.224849 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:49.224855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:49.224995 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:49.225124 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:49.229556 (+ 4432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.230830 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.234977 (+ 4147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.235914 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.236116 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:49.243111 (+ 6995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:49.243128 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:49.243185 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:49.244211 (+ 1026us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:49.244291 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:49.244326 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:49.244474 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:49.244817 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 11:31:49.244974 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 11:31:49.247344 10719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:49.247949 11028 ts_tablet_manager.cc:1918] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:31:49.248171 11028 log.cc:1196] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/0b0a261833424a748d3a211f8cbd76e2
I20260929 11:31:49.248762 11028 ts_tablet_manager.cc:1939] T 0b0a261833424a748d3a211f8cbd76e2 P e64630a2cbf84ef38a61319a0eb1aa0b: Deleting consensus metadata
I20260929 11:31:49.251096 10524 catalog_manager.cc:4841] TS e64630a2cbf84ef38a61319a0eb1aa0b (127.23.153.2:35073): tablet 0b0a261833424a748d3a211f8cbd76e2 (table test-table [id=b4d58355ba6d4b758832de85b3d6504c]) successfully deleted
I20260929 11:31:49.348846 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 10597
I20260929 11:31:49.349133 10523 catalog_manager.cc:4841] TS 32a6617dbc794f9e8c2dac4cf223b76d (127.23.153.1:37627): tablet 0b0a261833424a748d3a211f8cbd76e2 (table test-table [id=b4d58355ba6d4b758832de85b3d6504c]) successfully deleted
I20260929 11:31:49.364863 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 10728
I20260929 11:31:49.372381 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:49.365701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.23.153.2:35073)
0929 11:31:49.365889 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:49.365893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:49.365912 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:31:49.366031 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:49.366034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:49.366126 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:49.366210 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:49.366729 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.367689 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.369097 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.369646 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.369785 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:49.370913 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:49.370922 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:49.370937 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:49.371629 (+ 692us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:49.371683 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:49.371705 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:49.371800 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:49.371934 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 11:31:49.372048 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 11:31:49.373063 10850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:49.464419 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 10728
I20260929 11:31:49.478849 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 10859
I20260929 11:31:49.485921 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:49.479698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.23.153.3:44617)
0929 11:31:49.479894 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:49.479900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:49.479921 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:31:49.480115 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:49.480120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:49.480224 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:49.480311 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:49.480721 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.481459 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.482839 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:49.483344 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:49.483477 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:49.484579 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:49.484588 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:49.484604 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:49.485175 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:49.485225 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:49.485254 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:49.485372 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:49.485494 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 11:31:49.485615 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:31:49.486718 10981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:49.585234 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 10859
I20260929 11:31:49.600346 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 10507
I20260929 11:31:49.601449 10568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:49.720373 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 10507
2026-09-29T11:31:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2993 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:31:49 dist-test-slave-qk05 krb5kdc[11038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:31:49 dist-test-slave-qk05 krb5kdc[11038](info): set up 2 sockets
Sep 29 11:31:49 dist-test-slave-qk05 krb5kdc[11038](info): commencing operation
krb5kdc: starting...
W20260929 11:31:50.783676 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.011s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:31:50 dist-test-slave-qk05 krb5kdc[11038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:31:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:31:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:31:51.054876 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:40969
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:40969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:31:51.418134 11051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:51.418476 11051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:51.418596 11051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:51.426441 11051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:31:51.426842 11051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:51.427053 11051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:51.427331 11051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:31:51.427472 11051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:31:51.440763 11051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:40969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:40969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:51.442252 11051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:51.443804 11051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:51.459056 11059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:51.459578 11056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:51.459625 11057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:51.460530 11051 server_base.cc:1034] running on GCE node
I20260929 11:31:51.461953 11051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:51.464128 11051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:51.465437 11051 hybrid_clock.cc:648] HybridClock initialized: now 1790681511465377 us; error 52 us; skew 500 ppm
I20260929 11:31:51.465893 11051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:31:51 dist-test-slave-qk05 krb5kdc[11038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681511, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:31:51.469695 11051 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:31:51.471619 11051 webserver.cc:460] Webserver started at http://127.23.153.62:36237/ using document root <none> and password file <none>
I20260929 11:31:51.472491 11051 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:51.472613 11051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:51.472930 11051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:51.476661 11051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "f59719ae91084faeb8cb74a66d960e3a"
format_stamp: "Formatted at 2026-09-29 11:31:51 on dist-test-slave-qk05"
server_key: "acce4cede3933eeac57507c6602cb090"
server_key_iv: "76428b6b20c7636915b548663113f773"
server_key_version: "encryptionkey@0"
I20260929 11:31:51.477676 11051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "f59719ae91084faeb8cb74a66d960e3a"
format_stamp: "Formatted at 2026-09-29 11:31:51 on dist-test-slave-qk05"
server_key: "acce4cede3933eeac57507c6602cb090"
server_key_iv: "76428b6b20c7636915b548663113f773"
server_key_version: "encryptionkey@0"
I20260929 11:31:51.484197 11051 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260929 11:31:51.488404 11066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:51.490044 11051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:31:51.490257 11051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "f59719ae91084faeb8cb74a66d960e3a"
format_stamp: "Formatted at 2026-09-29 11:31:51 on dist-test-slave-qk05"
server_key: "acce4cede3933eeac57507c6602cb090"
server_key_iv: "76428b6b20c7636915b548663113f773"
server_key_version: "encryptionkey@0"
I20260929 11:31:51.490473 11051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:51.508615 11051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:51.514117 11051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:51.514441 11051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:51.535267 11051 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:40969
I20260929 11:31:51.535190 11117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:40969 every 8 connection(s)
I20260929 11:31:51.536739 11051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:31:51.541226 11118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:51.541791 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 11051
I20260929 11:31:51.541977 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:31:51.542586 24164 external_mini_cluster.cc:1428] Setting key 86e466c7c9b914c0ef5f2dec4a069aba
I20260929 11:31:51.557085 11118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a: Bootstrap starting.
Sep 29 11:31:51 dist-test-slave-qk05 krb5kdc[11038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:31:51.561398 11118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:51.562606 11118 log.cc:824] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:51.565790 11118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a: No bootstrap required, opened a new log
I20260929 11:31:51.570027 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:51.544672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45410 (local address 127.23.153.62:40969)
0929 11:31:51.545554 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:51.545567 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:51.545616 (+ 49us) server_negotiation.cc:407] Connection header received
0929 11:31:51.548244 (+ 2628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:51.548283 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:51.548831 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:51.549225 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:51.551199 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:51.552668 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:51.554704 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:51.555655 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:51.555838 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:51.561088 (+ 5250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:51.561098 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:51.561116 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:51.561164 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:51.564480 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:51.565477 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:51.565485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:51.565492 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:51.565796 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:51.566298 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:51.566302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:51.566307 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:51.566962 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:51.567272 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:51.568242 (+ 970us) server_negotiation.cc:299] Negotiation successful
0929 11:31:51.568791 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":409,"threads_started":1}
I20260929 11:31:51.575163 11118 raft_consensus.cc:348] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f59719ae91084faeb8cb74a66d960e3a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40969 } }
I20260929 11:31:51.575589 11118 raft_consensus.cc:374] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:51.575688 11118 raft_consensus.cc:729] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f59719ae91084faeb8cb74a66d960e3a, State: Initialized, Role: FOLLOWER
I20260929 11:31:51.576267 11118 consensus_queue.cc:260] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f59719ae91084faeb8cb74a66d960e3a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40969 } }
I20260929 11:31:51.576524 11118 raft_consensus.cc:388] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:31:51.576654 11118 raft_consensus.cc:482] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:31:51.576813 11118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:51.579465 11118 raft_consensus.cc:504] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f59719ae91084faeb8cb74a66d960e3a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40969 } }
I20260929 11:31:51.580050 11118 leader_election.cc:302] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f59719ae91084faeb8cb74a66d960e3a; no voters:
I20260929 11:31:51.580673 11118 leader_election.cc:288] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:31:51.580825 11123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:31:51.581494 11123 raft_consensus.cc:686] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a [term 1 LEADER]: Becoming Leader. State: Replica: f59719ae91084faeb8cb74a66d960e3a, State: Running, Role: LEADER
I20260929 11:31:51.582165 11123 consensus_queue.cc:237] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f59719ae91084faeb8cb74a66d960e3a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40969 } }
I20260929 11:31:51.582773 11118 sys_catalog.cc:564] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:31:51.584904 11124 sys_catalog.cc:455] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f59719ae91084faeb8cb74a66d960e3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f59719ae91084faeb8cb74a66d960e3a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40969 } } }
I20260929 11:31:51.585377 11124 sys_catalog.cc:458] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a [sys.catalog]: This master's current role is: LEADER
I20260929 11:31:51.585712 11125 sys_catalog.cc:455] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f59719ae91084faeb8cb74a66d960e3a. Latest consensus state: current_term: 1 leader_uuid: "f59719ae91084faeb8cb74a66d960e3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f59719ae91084faeb8cb74a66d960e3a" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40969 } } }
I20260929 11:31:51.586036 11125 sys_catalog.cc:458] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a [sys.catalog]: This master's current role is: LEADER
I20260929 11:31:51.589205 11129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:31:51.594914 11129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:31:51.606168 11129 catalog_manager.cc:1347] Generated new cluster ID: cd546a18defc4050bea8e198080a585c
I20260929 11:31:51.606290 11129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:31:51.619859 11129 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:31:51.621927 11129 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:31:51.635069 11129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a: Generated new TSK 0
I20260929 11:31:51.636196 11129 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.153.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
Entry for principal kudu/127.23.153.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab.
I20260929 11:31:51.716420 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40969
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:31:52.147710 11144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:52.148015 11144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:52.148152 11144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:52.155885 11144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:52.156031 11144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:52.156181 11144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:31:52.167742 11144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.1.keytab
--principal=kudu/127.23.153.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:52.169111 11144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:52.170756 11144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:52.184753 11152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:52.184736 11150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:52.185099 11149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:52.185537 11144 server_base.cc:1034] running on GCE node
I20260929 11:31:52.186232 11144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:52.187379 11144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:52.188663 11144 hybrid_clock.cc:648] HybridClock initialized: now 1790681512188603 us; error 44 us; skew 500 ppm
I20260929 11:31:52.189033 11144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:31:52 dist-test-slave-qk05 krb5kdc[11038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681512, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:31:52.192440 11144 init.cc:377] Logged in from keytab as kudu/127.23.153.1@KRBTEST.COM (short username kudu)
I20260929 11:31:52.193994 11144 webserver.cc:460] Webserver started at http://127.23.153.1:45061/ using document root <none> and password file <none>
I20260929 11:31:52.194773 11144 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:52.194907 11144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:52.195291 11144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:52.198784 11144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "407a228104714d34bb4209dd5efabc0f"
format_stamp: "Formatted at 2026-09-29 11:31:52 on dist-test-slave-qk05"
server_key: "00117eb5e2e305ef7682473ff5a05cbb"
server_key_iv: "af5b4b6508c3eb7d5d740fd0e6b7c5b9"
server_key_version: "encryptionkey@0"
I20260929 11:31:52.199755 11144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "407a228104714d34bb4209dd5efabc0f"
format_stamp: "Formatted at 2026-09-29 11:31:52 on dist-test-slave-qk05"
server_key: "00117eb5e2e305ef7682473ff5a05cbb"
server_key_iv: "af5b4b6508c3eb7d5d740fd0e6b7c5b9"
server_key_version: "encryptionkey@0"
I20260929 11:31:52.205832 11144 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:31:52.210335 11159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:52.212031 11144 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:31:52.212236 11144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "407a228104714d34bb4209dd5efabc0f"
format_stamp: "Formatted at 2026-09-29 11:31:52 on dist-test-slave-qk05"
server_key: "00117eb5e2e305ef7682473ff5a05cbb"
server_key_iv: "af5b4b6508c3eb7d5d740fd0e6b7c5b9"
server_key_version: "encryptionkey@0"
I20260929 11:31:52.212460 11144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:52.250025 11144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:52.255672 11144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:52.255982 11144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:52.257179 11144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:52.259655 11144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:52.259778 11144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:52.259913 11144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:52.259967 11144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:52.295279 11144 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:42629
I20260929 11:31:52.295325 11271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:42629 every 8 connection(s)
I20260929 11:31:52.297174 11144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:31:52.302148 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 11144
I20260929 11:31:52.302358 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:31:52.302945 24164 external_mini_cluster.cc:1428] Setting key 2a3b549fc8c92fc55ca86d15df8a7691
Sep 29 11:31:52 dist-test-slave-qk05 krb5kdc[11038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681512, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.1@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:31:52.322973 11274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:52.299992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:57567 (local address 127.23.153.62:40969)
0929 11:31:52.300462 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:52.300467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:52.301445 (+ 978us) server_negotiation.cc:407] Connection header received
0929 11:31:52.304093 (+ 2648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:52.304100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:52.304220 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:52.304326 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:52.306381 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:52.307240 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:52.309663 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:52.310125 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:52.310253 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:52.315384 (+ 5131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:52.315392 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:52.315398 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:52.315457 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:52.319376 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:52.320333 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:52.320338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:52.320342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:52.320520 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:52.320993 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:52.320997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:52.321000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:52.321346 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:52.321508 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:52.322321 (+ 813us) server_negotiation.cc:299] Negotiation successful
0929 11:31:52.322611 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":258,"threads_started":1}
I20260929 11:31:52.323822 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:52.300510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:40969 (local address 127.23.153.1:57567)
0929 11:31:52.301260 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:52.301309 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:52.303722 (+ 2413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:52.304641 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:52.304652 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:52.305251 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:31:52.306051 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:52.306101 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:52.307446 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:52.307452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:52.309412 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:52.309435 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:52.310397 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:52.310404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:52.310723 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:52.311570 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:31:52.311608 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:31:52.314952 (+ 3344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:31:52.319619 (+ 4667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:31:52.319628 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:31:52.319644 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:31:52.320197 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:31:52.320662 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:31:52.320666 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:31:52.320669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:31:52.320888 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:31:52.321465 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:31:52.321472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:31:52.321871 (+ 399us) client_negotiation.cc:769] Sending connection context
0929 11:31:52.322139 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 11:31:52.322591 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":289,"threads_started":1}
I20260929 11:31:52.325691 11272 heartbeater.cc:347] Connected to a master server at 127.23.153.62:40969
I20260929 11:31:52.326035 11272 heartbeater.cc:464] Registering TS with master...
I20260929 11:31:52.326859 11272 heartbeater.cc:511] Master 127.23.153.62:40969 requested a full tablet report, sending...
I20260929 11:31:52.329447 11083 ts_manager.cc:194] Registered new tserver with Master: 407a228104714d34bb4209dd5efabc0f (127.23.153.1:42629)
I20260929 11:31:52.331048 11083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.1@KRBTEST.COM'} at 127.23.153.1:57567
WARNING: no policy specified for kudu/127.23.153.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
Entry for principal kudu/127.23.153.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab.
I20260929 11:31:52.362541 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40969
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:31:52.733033 11278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:52.733379 11278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:52.733518 11278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:52.741603 11278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:52.741770 11278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:52.741935 11278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:31:52.752295 11278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.2.keytab
--principal=kudu/127.23.153.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:52.753700 11278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:52.755077 11278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:52.769769 11283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:52.769724 11284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:52.769727 11286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:52.770637 11278 server_base.cc:1034] running on GCE node
I20260929 11:31:52.771356 11278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:52.772468 11278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:52.773725 11278 hybrid_clock.cc:648] HybridClock initialized: now 1790681512773663 us; error 53 us; skew 500 ppm
I20260929 11:31:52.774097 11278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:31:52 dist-test-slave-qk05 krb5kdc[11038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681512, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:31:52.777673 11278 init.cc:377] Logged in from keytab as kudu/127.23.153.2@KRBTEST.COM (short username kudu)
I20260929 11:31:52.779443 11278 webserver.cc:460] Webserver started at http://127.23.153.2:36815/ using document root <none> and password file <none>
I20260929 11:31:52.780252 11278 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:52.780388 11278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:52.780719 11278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:52.784055 11278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "a13f026ab9fe43668b24d8bec81819f1"
format_stamp: "Formatted at 2026-09-29 11:31:52 on dist-test-slave-qk05"
server_key: "d7587180a7a9dda557f9502740f0b01b"
server_key_iv: "398a88943e2e369d6e9b7ae430884fa6"
server_key_version: "encryptionkey@0"
I20260929 11:31:52.784921 11278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "a13f026ab9fe43668b24d8bec81819f1"
format_stamp: "Formatted at 2026-09-29 11:31:52 on dist-test-slave-qk05"
server_key: "d7587180a7a9dda557f9502740f0b01b"
server_key_iv: "398a88943e2e369d6e9b7ae430884fa6"
server_key_version: "encryptionkey@0"
I20260929 11:31:52.791184 11278 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 11:31:52.795584 11293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:52.797212 11278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 11:31:52.797433 11278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "a13f026ab9fe43668b24d8bec81819f1"
format_stamp: "Formatted at 2026-09-29 11:31:52 on dist-test-slave-qk05"
server_key: "d7587180a7a9dda557f9502740f0b01b"
server_key_iv: "398a88943e2e369d6e9b7ae430884fa6"
server_key_version: "encryptionkey@0"
I20260929 11:31:52.797663 11278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:52.809821 11278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:52.815724 11278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:52.816079 11278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:52.817420 11278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:52.819267 11278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:52.819365 11278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:52.819500 11278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:52.819555 11278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:52.859272 11278 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:40947
I20260929 11:31:52.859324 11405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:40947 every 8 connection(s)
I20260929 11:31:52.860756 11278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:31:52.865017 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 11278
I20260929 11:31:52.865185 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:31:52.865675 24164 external_mini_cluster.cc:1428] Setting key fd725baa8d83f78f7dd37a0d6ada9a31
Sep 29 11:31:52 dist-test-slave-qk05 krb5kdc[11038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681512, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.2@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:31:52.888540 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:52.864321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:48427 (local address 127.23.153.62:40969)
0929 11:31:52.864892 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:52.864899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:52.866110 (+ 1211us) server_negotiation.cc:407] Connection header received
0929 11:31:52.868685 (+ 2575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:52.868691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:52.868782 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:52.868900 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:52.870815 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:52.872293 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:52.874801 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:52.875537 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:52.875718 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:52.879969 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:52.879976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:52.879979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:52.880023 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:52.883088 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:52.884336 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:52.884342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:52.884344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:52.884510 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:52.885304 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:52.885313 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:52.885315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:52.885980 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:52.886270 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:52.887776 (+ 1506us) server_negotiation.cc:299] Negotiation successful
0929 11:31:52.888081 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":279,"threads_started":1}
I20260929 11:31:52.890450 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:52.864982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:40969 (local address 127.23.153.2:48427)
0929 11:31:52.865890 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:52.865943 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:52.868336 (+ 2393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:52.869158 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:52.869167 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:52.869880 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:31:52.870586 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:52.870610 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:52.872657 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:52.872663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:52.874380 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:52.874416 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:52.875899 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:52.875906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:52.876169 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:52.876965 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:31:52.876993 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:31:52.879617 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:31:52.883361 (+ 3744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:31:52.883373 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:31:52.883397 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:31:52.884154 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:31:52.884718 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:31:52.884722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:31:52.884726 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:31:52.885063 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:31:52.886253 (+ 1190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:31:52.886262 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:31:52.886895 (+ 633us) client_negotiation.cc:769] Sending connection context
0929 11:31:52.887542 (+ 647us) client_negotiation.cc:240] Negotiation successful
0929 11:31:52.888366 (+ 824us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":345,"threads_started":1}
I20260929 11:31:52.892679 11406 heartbeater.cc:347] Connected to a master server at 127.23.153.62:40969
I20260929 11:31:52.893307 11406 heartbeater.cc:464] Registering TS with master...
I20260929 11:31:52.894466 11406 heartbeater.cc:511] Master 127.23.153.62:40969 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.153.3@KRBTEST.COM; defaulting to no policy
I20260929 11:31:52.896878 11083 ts_manager.cc:194] Registered new tserver with Master: a13f026ab9fe43668b24d8bec81819f1 (127.23.153.2:40947)
I20260929 11:31:52.897886 11083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.2@KRBTEST.COM'} at 127.23.153.2:48427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
Entry for principal kudu/127.23.153.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab.
I20260929 11:31:52.931180 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40969
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:31:53.303242 11412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:53.303553 11412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:53.303654 11412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:53.311327 11412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:53.311468 11412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:53.311599 11412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:31:53.322422 11412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.3.keytab
--principal=kudu/127.23.153.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:40969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:53.323855 11412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:53.325161 11412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 11:31:53.334587 11272 heartbeater.cc:503] Master 127.23.153.62:40969 was elected leader, sending a full tablet report...
W20260929 11:31:53.339003 11418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:53.338989 11420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:53.339251 11417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:53.339946 11412 server_base.cc:1034] running on GCE node
I20260929 11:31:53.340651 11412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:53.341747 11412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:53.343005 11412 hybrid_clock.cc:648] HybridClock initialized: now 1790681513342946 us; error 51 us; skew 500 ppm
I20260929 11:31:53.343401 11412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:31:53 dist-test-slave-qk05 krb5kdc[11038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681513, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:31:53.347460 11412 init.cc:377] Logged in from keytab as kudu/127.23.153.3@KRBTEST.COM (short username kudu)
I20260929 11:31:53.349104 11412 webserver.cc:460] Webserver started at http://127.23.153.3:43351/ using document root <none> and password file <none>
I20260929 11:31:53.349843 11412 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:53.349973 11412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:53.350306 11412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:53.353578 11412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "ebac9755b59a43e99027df4ffc4ae930"
format_stamp: "Formatted at 2026-09-29 11:31:53 on dist-test-slave-qk05"
server_key: "9bd08ef203d70586ea5ce4dee860d8bb"
server_key_iv: "a4146dc4e36ea54e6b570ac743be7da4"
server_key_version: "encryptionkey@0"
I20260929 11:31:53.354465 11412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "ebac9755b59a43e99027df4ffc4ae930"
format_stamp: "Formatted at 2026-09-29 11:31:53 on dist-test-slave-qk05"
server_key: "9bd08ef203d70586ea5ce4dee860d8bb"
server_key_iv: "a4146dc4e36ea54e6b570ac743be7da4"
server_key_version: "encryptionkey@0"
I20260929 11:31:53.361629 11412 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260929 11:31:53.365983 11427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:53.367686 11412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:31:53.367888 11412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "ebac9755b59a43e99027df4ffc4ae930"
format_stamp: "Formatted at 2026-09-29 11:31:53 on dist-test-slave-qk05"
server_key: "9bd08ef203d70586ea5ce4dee860d8bb"
server_key_iv: "a4146dc4e36ea54e6b570ac743be7da4"
server_key_version: "encryptionkey@0"
I20260929 11:31:53.368131 11412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:53.405259 11412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:53.410301 11412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:53.410589 11412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:53.411612 11412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:53.413285 11412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:53.413375 11412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:53.413492 11412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:53.413753 11412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:53.447984 11412 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:40743
I20260929 11:31:53.448031 11539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:40743 every 8 connection(s)
I20260929 11:31:53.449339 11412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:31:53.459589 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 11412
I20260929 11:31:53.460031 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:31:53.460721 24164 external_mini_cluster.cc:1428] Setting key b1faa4d829fd2facc076cef4c24af291
Sep 29 11:31:53 dist-test-slave-qk05 krb5kdc[11038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681513, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.3@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:31:53.474709 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:53.452395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:59005 (local address 127.23.153.62:40969)
0929 11:31:53.453098 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:53.453103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:53.453762 (+ 659us) server_negotiation.cc:407] Connection header received
0929 11:31:53.455941 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:53.455948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:53.456060 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:53.456221 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:53.458212 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.459289 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.462151 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.462851 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.462991 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:53.467337 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:53.467343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:53.467347 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:53.467389 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:53.470180 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:53.471185 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:53.471190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:53.471192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:53.471341 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:53.471905 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:53.471909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:53.471912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:53.472281 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:53.472424 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:53.473977 (+ 1553us) server_negotiation.cc:299] Negotiation successful
0929 11:31:53.474324 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":420,"threads_started":1}
I20260929 11:31:53.475922 11543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:53.452682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:40969 (local address 127.23.153.3:59005)
0929 11:31:53.453550 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:53.453600 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:53.455497 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:53.456581 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:53.456596 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:53.457228 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:31:53.457910 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:53.457937 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.459635 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.459644 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:53.461935 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:53.461962 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.463203 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.463213 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:53.463458 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:53.464187 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:31:53.464221 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:31:53.466985 (+ 2764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:31:53.470399 (+ 3414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:31:53.470409 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:31:53.470427 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:31:53.470981 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:31:53.471504 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:31:53.471508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:31:53.471511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:31:53.471746 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:31:53.472414 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:31:53.472421 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:31:53.473318 (+ 897us) client_negotiation.cc:769] Sending connection context
0929 11:31:53.473664 (+ 346us) client_negotiation.cc:240] Negotiation successful
0929 11:31:53.474156 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":346,"threads_started":1}
I20260929 11:31:53.477629 11540 heartbeater.cc:347] Connected to a master server at 127.23.153.62:40969
I20260929 11:31:53.477957 11540 heartbeater.cc:464] Registering TS with master...
I20260929 11:31:53.478713 11540 heartbeater.cc:511] Master 127.23.153.62:40969 requested a full tablet report, sending...
I20260929 11:31:53.480430 11083 ts_manager.cc:194] Registered new tserver with Master: ebac9755b59a43e99027df4ffc4ae930 (127.23.153.3:40743)
I20260929 11:31:53.481280 11083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.153.3@KRBTEST.COM'} at 127.23.153.3:59005
I20260929 11:31:53.490419 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:31:53.510058 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:53.495422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45418 (local address 127.23.153.62:40969)
0929 11:31:53.495718 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:53.495725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:53.496220 (+ 495us) server_negotiation.cc:407] Connection header received
0929 11:31:53.496374 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:53.496379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:53.496456 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:53.496565 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:53.498056 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.498877 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.500472 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.500887 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.501000 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:53.503668 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:53.503674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:53.503678 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:53.503722 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:53.506491 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:53.507435 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:53.507440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:53.507443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:53.507604 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:53.508088 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:53.508093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:53.508095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:53.508467 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:53.508696 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:53.509442 (+ 746us) server_negotiation.cc:299] Negotiation successful
0929 11:31:53.509765 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 11:31:53.515079 11083 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:31:53.522874 11083 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:31:53.571259 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:53.539570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.23.153.1:42629)
0929 11:31:53.560670 (+ 21100us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:53.560682 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:53.560724 (+ 42us) server_negotiation.cc:407] Connection header received
0929 11:31:53.560858 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:53.560867 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:53.561219 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:53.562765 (+ 1546us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:53.563060 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.564607 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.568196 (+ 3589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.569843 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.570003 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:53.570248 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:53.570665 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 11:31:53.570859 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20851,"thread_start_us":21874,"threads_started":1}
I20260929 11:31:53.571309 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:53.538875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.23.153.2:40947)
0929 11:31:53.557069 (+ 18194us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:53.557079 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:53.557113 (+ 34us) server_negotiation.cc:407] Connection header received
0929 11:31:53.557316 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:53.557330 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:53.557698 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:53.557985 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:53.564052 (+ 6067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.565718 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.568289 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.569849 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.570007 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:53.570216 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:53.570779 (+ 563us) server_negotiation.cc:299] Negotiation successful
0929 11:31:53.570966 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18002,"thread_start_us":17869,"threads_started":1}
I20260929 11:31:53.571413 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:53.539224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:42629 (local address 127.0.0.1:58950)
0929 11:31:53.556414 (+ 17190us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:53.556430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:53.556532 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:53.561852 (+ 5320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:53.561862 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:53.561926 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:53.562454 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:53.562471 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.564775 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.564781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:53.567705 (+ 2924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:53.567721 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.570090 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.570098 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:53.570368 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:53.570429 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 11:31:53.570581 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 11:31:53.570809 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17085,"mutex_wait_us":88,"thread_start_us":15983,"threads_started":1}
I20260929 11:31:53.572234 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:53.538855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:40947 (local address 127.0.0.1:52470)
0929 11:31:53.555674 (+ 16819us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:53.555720 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:53.555960 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:53.563150 (+ 7190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:53.563156 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:53.563194 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:53.563742 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:53.563758 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.565906 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.565910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:53.568067 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:53.568083 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.570390 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.570395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:53.570567 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:53.570601 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 11:31:53.571600 (+ 999us) client_negotiation.cc:240] Negotiation successful
0929 11:31:53.571765 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16505,"mutex_wait_us":141,"thread_start_us":17336,"threads_started":1}
I20260929 11:31:53.575222 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:53.557154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:40743 (local address 127.0.0.1:36676)
0929 11:31:53.563863 (+ 6709us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:53.563886 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:53.564029 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:53.566639 (+ 2610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:53.566645 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:53.566677 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:53.567186 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:53.567204 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.569459 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.569475 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:53.571501 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:53.571522 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.574352 (+ 2830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.574360 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:53.574508 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:53.574534 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 11:31:53.574616 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 11:31:53.574771 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6540,"mutex_wait_us":1,"thread_start_us":6953,"threads_started":1}
I20260929 11:31:53.575291 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:53.557147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36676 (local address 127.23.153.3:40743)
0929 11:31:53.565779 (+ 8632us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:53.565789 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:53.565834 (+ 45us) server_negotiation.cc:407] Connection header received
0929 11:31:53.565954 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:53.565963 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:53.566317 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:53.566637 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:53.567425 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.569322 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.571732 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.573770 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.573913 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:53.574175 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:53.574734 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 11:31:53.574919 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8224,"thread_start_us":8089,"threads_started":1}
I20260929 11:31:53.576056 11341 tablet_service.cc:1467] Processing CreateTablet for tablet 95705f2176c04b5e942ef6225919d46d (DEFAULT_TABLE table=test-table [id=110a5e6d0c1a473c8373857b1cbfd3dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:31:53.576056 11207 tablet_service.cc:1467] Processing CreateTablet for tablet 95705f2176c04b5e942ef6225919d46d (DEFAULT_TABLE table=test-table [id=110a5e6d0c1a473c8373857b1cbfd3dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:31:53.577719 11341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95705f2176c04b5e942ef6225919d46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:53.577723 11207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95705f2176c04b5e942ef6225919d46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:53.579335 11475 tablet_service.cc:1467] Processing CreateTablet for tablet 95705f2176c04b5e942ef6225919d46d (DEFAULT_TABLE table=test-table [id=110a5e6d0c1a473c8373857b1cbfd3dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:31:53.580986 11475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95705f2176c04b5e942ef6225919d46d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:53.595896 11561 tablet_bootstrap.cc:492] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930: Bootstrap starting.
I20260929 11:31:53.599550 11561 tablet_bootstrap.cc:654] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:53.599956 11560 tablet_bootstrap.cc:492] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1: Bootstrap starting.
I20260929 11:31:53.601221 11561 log.cc:824] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:53.601907 11559 tablet_bootstrap.cc:492] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f: Bootstrap starting.
I20260929 11:31:53.606251 11559 tablet_bootstrap.cc:654] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:53.606691 11561 tablet_bootstrap.cc:492] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930: No bootstrap required, opened a new log
I20260929 11:31:53.606726 11560 tablet_bootstrap.cc:654] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:53.607080 11561 ts_tablet_manager.cc:1397] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260929 11:31:53.607779 11559 log.cc:824] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:53.607867 11560 log.cc:824] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:53.611593 11559 tablet_bootstrap.cc:492] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f: No bootstrap required, opened a new log
I20260929 11:31:53.611641 11560 tablet_bootstrap.cc:492] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1: No bootstrap required, opened a new log
I20260929 11:31:53.611927 11559 ts_tablet_manager.cc:1397] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260929 11:31:53.611968 11560 ts_tablet_manager.cc:1397] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260929 11:31:53.618366 11561 raft_consensus.cc:348] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13f026ab9fe43668b24d8bec81819f1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 40947 } } peers { permanent_uuid: "ebac9755b59a43e99027df4ffc4ae930" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40743 } } peers { permanent_uuid: "407a228104714d34bb4209dd5efabc0f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 42629 } }
I20260929 11:31:53.618974 11561 raft_consensus.cc:374] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:53.619114 11561 raft_consensus.cc:729] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebac9755b59a43e99027df4ffc4ae930, State: Initialized, Role: FOLLOWER
I20260929 11:31:53.620306 11561 consensus_queue.cc:260] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13f026ab9fe43668b24d8bec81819f1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 40947 } } peers { permanent_uuid: "ebac9755b59a43e99027df4ffc4ae930" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40743 } } peers { permanent_uuid: "407a228104714d34bb4209dd5efabc0f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 42629 } }
I20260929 11:31:53.620949 11559 raft_consensus.cc:348] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13f026ab9fe43668b24d8bec81819f1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 40947 } } peers { permanent_uuid: "ebac9755b59a43e99027df4ffc4ae930" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40743 } } peers { permanent_uuid: "407a228104714d34bb4209dd5efabc0f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 42629 } }
I20260929 11:31:53.621461 11559 raft_consensus.cc:374] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:53.621564 11559 raft_consensus.cc:729] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 407a228104714d34bb4209dd5efabc0f, State: Initialized, Role: FOLLOWER
I20260929 11:31:53.621851 11540 heartbeater.cc:503] Master 127.23.153.62:40969 was elected leader, sending a full tablet report...
I20260929 11:31:53.622253 11559 consensus_queue.cc:260] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13f026ab9fe43668b24d8bec81819f1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 40947 } } peers { permanent_uuid: "ebac9755b59a43e99027df4ffc4ae930" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40743 } } peers { permanent_uuid: "407a228104714d34bb4209dd5efabc0f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 42629 } }
I20260929 11:31:53.622532 11561 ts_tablet_manager.cc:1428] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260929 11:31:53.624233 11559 ts_tablet_manager.cc:1428] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260929 11:31:53.624670 11560 raft_consensus.cc:348] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13f026ab9fe43668b24d8bec81819f1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 40947 } } peers { permanent_uuid: "ebac9755b59a43e99027df4ffc4ae930" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40743 } } peers { permanent_uuid: "407a228104714d34bb4209dd5efabc0f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 42629 } }
I20260929 11:31:53.625152 11560 raft_consensus.cc:374] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:53.625258 11560 raft_consensus.cc:729] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a13f026ab9fe43668b24d8bec81819f1, State: Initialized, Role: FOLLOWER
I20260929 11:31:53.625975 11560 consensus_queue.cc:260] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13f026ab9fe43668b24d8bec81819f1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 40947 } } peers { permanent_uuid: "ebac9755b59a43e99027df4ffc4ae930" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40743 } } peers { permanent_uuid: "407a228104714d34bb4209dd5efabc0f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 42629 } }
I20260929 11:31:53.627279 11406 heartbeater.cc:503] Master 127.23.153.62:40969 was elected leader, sending a full tablet report...
I20260929 11:31:53.628588 11560 ts_tablet_manager.cc:1428] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
W20260929 11:31:53.701076 11541 tablet.cc:2369] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:31:53.799950 11567 raft_consensus.cc:482] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 11:31:53.800100 11273 tablet.cc:2369] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:31:53.800318 11567 raft_consensus.cc:504] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13f026ab9fe43668b24d8bec81819f1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 40947 } } peers { permanent_uuid: "ebac9755b59a43e99027df4ffc4ae930" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40743 } } peers { permanent_uuid: "407a228104714d34bb4209dd5efabc0f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 42629 } }
I20260929 11:31:53.802174 11567 leader_election.cc:288] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebac9755b59a43e99027df4ffc4ae930 (127.23.153.3:40743), 407a228104714d34bb4209dd5efabc0f (127.23.153.1:42629)
I20260929 11:31:53.810921 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:53.802728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:46753 (local address 127.23.153.1:42629)
0929 11:31:53.802918 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:53.802922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:53.803788 (+ 866us) server_negotiation.cc:407] Connection header received
0929 11:31:53.804082 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:53.804088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:53.804169 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:53.804233 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:53.804987 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.805830 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.808321 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.809891 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.810051 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:53.810113 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:53.810558 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 11:31:53.810674 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 11:31:53.811115 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:53.802557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:42629 (local address 127.23.153.2:46753)
0929 11:31:53.803667 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:53.803688 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:53.803912 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:53.804451 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:53.804456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:53.804484 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:53.804839 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:53.804852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.806093 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.806102 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:53.808087 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:53.808107 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.810165 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.810173 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:53.810355 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:53.810394 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 11:31:53.810532 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 11:31:53.810700 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"thread_start_us":1002,"threads_started":1}
I20260929 11:31:53.811898 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:53.802548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:46317 (local address 127.23.153.3:40743)
0929 11:31:53.802791 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:53.802799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:53.804577 (+ 1778us) server_negotiation.cc:407] Connection header received
0929 11:31:53.804783 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:53.804788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:53.804868 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:53.804952 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:53.805510 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.806874 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.809606 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.810730 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.810920 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:53.811005 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:53.811510 (+ 505us) server_negotiation.cc:299] Negotiation successful
0929 11:31:53.811657 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 11:31:53.811936 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:53.803283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:40743 (local address 127.23.153.2:46317)
0929 11:31:53.804441 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:53.804460 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:53.804638 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:53.805014 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:53.805018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:53.805042 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:53.805368 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:53.805379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.807119 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.807124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:53.809300 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:53.809318 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.811042 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.811050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:53.811271 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:53.811305 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 11:31:53.811451 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 11:31:53.811603 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"thread_start_us":827,"threads_started":1}
I20260929 11:31:53.812440 11227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95705f2176c04b5e942ef6225919d46d" candidate_uuid: "a13f026ab9fe43668b24d8bec81819f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "407a228104714d34bb4209dd5efabc0f" is_pre_election: true
I20260929 11:31:53.812981 11227 raft_consensus.cc:2454] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a13f026ab9fe43668b24d8bec81819f1 in term 0.
I20260929 11:31:53.812897 11495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95705f2176c04b5e942ef6225919d46d" candidate_uuid: "a13f026ab9fe43668b24d8bec81819f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebac9755b59a43e99027df4ffc4ae930" is_pre_election: true
I20260929 11:31:53.813339 11495 raft_consensus.cc:2454] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a13f026ab9fe43668b24d8bec81819f1 in term 0.
I20260929 11:31:53.813755 11295 leader_election.cc:302] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 407a228104714d34bb4209dd5efabc0f, a13f026ab9fe43668b24d8bec81819f1; no voters:
I20260929 11:31:53.814261 11567 raft_consensus.cc:2781] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:31:53.814432 11567 raft_consensus.cc:482] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:31:53.814538 11567 raft_consensus.cc:3037] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:53.817129 11567 raft_consensus.cc:504] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13f026ab9fe43668b24d8bec81819f1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 40947 } } peers { permanent_uuid: "ebac9755b59a43e99027df4ffc4ae930" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40743 } } peers { permanent_uuid: "407a228104714d34bb4209dd5efabc0f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 42629 } }
I20260929 11:31:53.817838 11567 leader_election.cc:288] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [CANDIDATE]: Term 1 election: Requested vote from peers ebac9755b59a43e99027df4ffc4ae930 (127.23.153.3:40743), 407a228104714d34bb4209dd5efabc0f (127.23.153.1:42629)
I20260929 11:31:53.818544 11227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95705f2176c04b5e942ef6225919d46d" candidate_uuid: "a13f026ab9fe43668b24d8bec81819f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "407a228104714d34bb4209dd5efabc0f"
I20260929 11:31:53.818524 11495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95705f2176c04b5e942ef6225919d46d" candidate_uuid: "a13f026ab9fe43668b24d8bec81819f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebac9755b59a43e99027df4ffc4ae930"
I20260929 11:31:53.818794 11495 raft_consensus.cc:3037] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:53.818794 11227 raft_consensus.cc:3037] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:53.822285 11227 raft_consensus.cc:2454] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a13f026ab9fe43668b24d8bec81819f1 in term 1.
I20260929 11:31:53.822285 11495 raft_consensus.cc:2454] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a13f026ab9fe43668b24d8bec81819f1 in term 1.
I20260929 11:31:53.822960 11295 leader_election.cc:302] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 407a228104714d34bb4209dd5efabc0f, a13f026ab9fe43668b24d8bec81819f1, ebac9755b59a43e99027df4ffc4ae930; no voters:
I20260929 11:31:53.823431 11567 raft_consensus.cc:2781] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:31:53.823930 11567 raft_consensus.cc:686] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [term 1 LEADER]: Becoming Leader. State: Replica: a13f026ab9fe43668b24d8bec81819f1, State: Running, Role: LEADER
I20260929 11:31:53.824445 11567 consensus_queue.cc:237] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13f026ab9fe43668b24d8bec81819f1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 40947 } } peers { permanent_uuid: "ebac9755b59a43e99027df4ffc4ae930" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40743 } } peers { permanent_uuid: "407a228104714d34bb4209dd5efabc0f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 42629 } }
I20260929 11:31:53.829708 11080 catalog_manager.cc:5495] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a13f026ab9fe43668b24d8bec81819f1 (127.23.153.2). New cstate: current_term: 1 leader_uuid: "a13f026ab9fe43668b24d8bec81819f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a13f026ab9fe43668b24d8bec81819f1" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 40947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebac9755b59a43e99027df4ffc4ae930" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "407a228104714d34bb4209dd5efabc0f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 42629 } health_report { overall_health: UNKNOWN } } }
W20260929 11:31:53.863041 11407 tablet.cc:2369] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:31:53.877072 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:53.870319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52482 (local address 127.23.153.2:40947)
0929 11:31:53.870651 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:53.870656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:53.870750 (+ 94us) server_negotiation.cc:407] Connection header received
0929 11:31:53.870967 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:53.870971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:53.871105 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:53.871255 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:31:53.871705 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.872695 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.874325 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.874905 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.875071 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:53.875716 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:31:53.876301 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:31:53.876425 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:53.876692 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 11:31:53.876773 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 11:31:53.887439 11495 raft_consensus.cc:1261] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930 [term 1 FOLLOWER]: Refusing update from remote peer a13f026ab9fe43668b24d8bec81819f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:31:53.887439 11227 raft_consensus.cc:1261] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f [term 1 FOLLOWER]: Refusing update from remote peer a13f026ab9fe43668b24d8bec81819f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:31:53.888957 11570 consensus_queue.cc:1035] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebac9755b59a43e99027df4ffc4ae930" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 40743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:31:53.889379 11567 consensus_queue.cc:1035] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "407a228104714d34bb4209dd5efabc0f" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 42629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:31:53.920142 11080 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:31:53.920436 11080 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:31:53.924608 11080 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f59719ae91084faeb8cb74a66d960e3a: Sending DeleteTablet for 3 replicas of tablet 95705f2176c04b5e942ef6225919d46d
I20260929 11:31:53.926023 11475 tablet_service.cc:1514] Processing DeleteTablet for tablet 95705f2176c04b5e942ef6225919d46d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:53 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:36676
I20260929 11:31:53.926191 11207 tablet_service.cc:1514] Processing DeleteTablet for tablet 95705f2176c04b5e942ef6225919d46d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:53 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:58950
I20260929 11:31:53.926193 11341 tablet_service.cc:1514] Processing DeleteTablet for tablet 95705f2176c04b5e942ef6225919d46d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:53 UTC) from {username='kudu', principal='kudu/127.23.153.62@KRBTEST.COM'} at 127.0.0.1:52470
I20260929 11:31:53.929023 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 11144
I20260929 11:31:53.938561 11585 tablet_replica.cc:330] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f: stopping tablet replica
I20260929 11:31:53.939144 11585 raft_consensus.cc:2229] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:31:53.939759 11585 raft_consensus.cc:2258] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:31:53.941249 11583 tablet_replica.cc:330] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930: stopping tablet replica
I20260929 11:31:53.941736 11583 raft_consensus.cc:2229] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:31:53.941865 11584 tablet_replica.cc:330] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1: stopping tablet replica
I20260929 11:31:53.942204 11583 raft_consensus.cc:2258] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:31:53.942306 11584 raft_consensus.cc:2229] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:31:53.942991 11584 raft_consensus.cc:2258] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:31:53.944061 11585 ts_tablet_manager.cc:1905] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:31:53.945348 11584 ts_tablet_manager.cc:1905] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:31:53.952148 11585 ts_tablet_manager.cc:1918] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:31:53.952378 11585 log.cc:1196] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-0/wal/wals/95705f2176c04b5e942ef6225919d46d
Sep 29 11:31:53 dist-test-slave-qk05 krb5kdc[11038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.1@KRBTEST.COM
I20260929 11:31:53.952934 11585 ts_tablet_manager.cc:1939] T 95705f2176c04b5e942ef6225919d46d P 407a228104714d34bb4209dd5efabc0f: Deleting consensus metadata
I20260929 11:31:53.953739 11584 ts_tablet_manager.cc:1918] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:31:53.953933 11584 log.cc:1196] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/95705f2176c04b5e942ef6225919d46d
I20260929 11:31:53.954463 11584 ts_tablet_manager.cc:1939] T 95705f2176c04b5e942ef6225919d46d P a13f026ab9fe43668b24d8bec81819f1: Deleting consensus metadata
I20260929 11:31:53.954433 11583 ts_tablet_manager.cc:1905] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:31:53.955114 11068 catalog_manager.cc:4841] TS 407a228104714d34bb4209dd5efabc0f (127.23.153.1:42629): tablet 95705f2176c04b5e942ef6225919d46d (table test-table [id=110a5e6d0c1a473c8373857b1cbfd3dc]) successfully deleted
I20260929 11:31:53.956588 11070 catalog_manager.cc:4841] TS a13f026ab9fe43668b24d8bec81819f1 (127.23.153.2:40947): tablet 95705f2176c04b5e942ef6225919d46d (table test-table [id=110a5e6d0c1a473c8373857b1cbfd3dc]) successfully deleted
I20260929 11:31:53.960640 11583 ts_tablet_manager.cc:1918] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:31:53.960819 11583 log.cc:1196] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/95705f2176c04b5e942ef6225919d46d
I20260929 11:31:53.961247 11583 ts_tablet_manager.cc:1939] T 95705f2176c04b5e942ef6225919d46d P ebac9755b59a43e99027df4ffc4ae930: Deleting consensus metadata
I20260929 11:31:53.963152 11070 catalog_manager.cc:4841] TS ebac9755b59a43e99027df4ffc4ae930 (127.23.153.3:40743): tablet 95705f2176c04b5e942ef6225919d46d (table test-table [id=110a5e6d0c1a473c8373857b1cbfd3dc]) successfully deleted
I20260929 11:31:53.965067 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:53.930882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58952 (local address 127.23.153.1:42629)
0929 11:31:53.931157 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:53.931166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:53.941163 (+ 9997us) server_negotiation.cc:407] Connection header received
0929 11:31:53.941392 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:53.941399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:53.941492 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:53.941600 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:53.943515 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.944688 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.946758 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:53.947443 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:53.947576 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:53.957015 (+ 9439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:53.957035 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:53.957054 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:53.957116 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:53.961236 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:53.962315 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:53.962325 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:53.962329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:53.962533 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:53.963074 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:53.963079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:53.963082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:53.963563 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:53.963821 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:53.964378 (+ 557us) server_negotiation.cc:299] Negotiation successful
0929 11:31:53.964657 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 11:31:53.965785 11267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:54.060853 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 11144
I20260929 11:31:54.078163 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 11278
Sep 29 11:31:54 dist-test-slave-qk05 krb5kdc[11038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.2@KRBTEST.COM
I20260929 11:31:54.095480 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:54.078921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52488 (local address 127.23.153.2:40947)
0929 11:31:54.079151 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:54.079161 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:54.079183 (+ 22us) server_negotiation.cc:407] Connection header received
0929 11:31:54.079289 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:54.079295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:54.079391 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:54.079477 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:54.080740 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:54.081596 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:54.082954 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:54.083558 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:54.083679 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:54.088439 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:54.088459 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:54.088476 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:54.088535 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:54.092001 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:54.093027 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:54.093036 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:54.093039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:54.093227 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:54.093642 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:54.093646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:54.093652 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:54.094103 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:54.094466 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:54.094765 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 11:31:54.095063 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 11:31:54.096328 11401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:54.213961 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 11278
I20260929 11:31:54.233445 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 11412
Sep 29 11:31:54 dist-test-slave-qk05 krb5kdc[11038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.3@KRBTEST.COM
I20260929 11:31:54.251413 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:54.234565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.23.153.3:40743)
0929 11:31:54.234793 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:54.234798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:54.234821 (+ 23us) server_negotiation.cc:407] Connection header received
0929 11:31:54.234989 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:54.234994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:54.235099 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:54.235207 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:54.236663 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:54.237437 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:54.238912 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:54.239385 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:54.239504 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:54.244202 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:54.244221 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:54.244238 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:54.244282 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:54.247397 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:54.248435 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:54.248444 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:54.248446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:54.248722 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:54.249234 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:54.249238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:54.249241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:54.249755 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:54.249997 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:54.250693 (+ 696us) server_negotiation.cc:299] Negotiation successful
0929 11:31:54.250996 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 11:31:54.252108 11535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:54.344980 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 11412
I20260929 11:31:54.359596 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 11051
I20260929 11:31:54.360827 11113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:54.477757 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 11051
2026-09-29T11:31:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4754 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T11:31:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:31:54Z Disabled control of system clock
I20260929 11:31:54.555577 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:34091
--webserver_interface=127.23.153.62
--webserver_port=0
--builtin_ntp_servers=127.23.153.20:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:34091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 11:31:54.925711 11594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:54.926059 11594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:54.926151 11594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:54.933988 11594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:31:54.934126 11594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:54.934163 11594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:54.934212 11594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:31:54.934260 11594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:31:54.944926 11594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:34091
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:34091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:54.946519 11594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:54.948626 11594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:54.961633 11602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:54.961753 11599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:54.961753 11600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:54.962579 11594 server_base.cc:1034] running on GCE node
I20260929 11:31:54.963876 11594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:54.965687 11594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:54.967084 11594 hybrid_clock.cc:648] HybridClock initialized: now 1790681514966880 us; error 127 us; skew 500 ppm
I20260929 11:31:54.967664 11594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:31:54.978116 11594 webserver.cc:460] Webserver started at http://127.23.153.62:39693/ using document root <none> and password file <none>
I20260929 11:31:54.978955 11594 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:54.979138 11594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:54.979485 11594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:54.982689 11594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "a994fba1c5a84a149b8f98f636d29294"
format_stamp: "Formatted at 2026-09-29 11:31:54 on dist-test-slave-qk05"
server_key: "da659e7304741f080e53c5b38f2dce91"
server_key_iv: "dd7ae70c1ffd40d7af78079e9b06c886"
server_key_version: "encryptionkey@0"
I20260929 11:31:54.983632 11594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "a994fba1c5a84a149b8f98f636d29294"
format_stamp: "Formatted at 2026-09-29 11:31:54 on dist-test-slave-qk05"
server_key: "da659e7304741f080e53c5b38f2dce91"
server_key_iv: "dd7ae70c1ffd40d7af78079e9b06c886"
server_key_version: "encryptionkey@0"
I20260929 11:31:54.989423 11594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 11:31:54.993521 11608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:54.995162 11594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:31:54.995376 11594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "a994fba1c5a84a149b8f98f636d29294"
format_stamp: "Formatted at 2026-09-29 11:31:54 on dist-test-slave-qk05"
server_key: "da659e7304741f080e53c5b38f2dce91"
server_key_iv: "dd7ae70c1ffd40d7af78079e9b06c886"
server_key_version: "encryptionkey@0"
I20260929 11:31:54.995637 11594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:55.018364 11594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:55.019397 11594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:55.019709 11594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:55.038847 11594 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:34091
I20260929 11:31:55.038888 11659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:34091 every 8 connection(s)
I20260929 11:31:55.040314 11594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:31:55.043213 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 11594
I20260929 11:31:55.043378 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:31:55.043816 24164 external_mini_cluster.cc:1428] Setting key f04fb4592e5e35222479ef99a507e4bb
I20260929 11:31:55.044653 11660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:55.061844 11660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294: Bootstrap starting.
I20260929 11:31:55.061846 11663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:55.045880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48482 (local address 127.23.153.62:34091)
0929 11:31:55.047135 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:55.047150 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:55.047206 (+ 56us) server_negotiation.cc:407] Connection header received
0929 11:31:55.049687 (+ 2481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:55.049729 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:55.050684 (+ 955us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:55.051298 (+ 614us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:55.051979 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:55.054035 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:55.055790 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:55.057026 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:55.057246 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:55.058479 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:55.058491 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:55.058595 (+ 104us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:55.059470 (+ 875us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:55.059607 (+ 137us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:55.059645 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:55.059773 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:55.060070 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 11:31:55.060460 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":385,"threads_started":1}
I20260929 11:31:55.066362 11660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:55.067734 11660 log.cc:824] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:55.072427 11660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294: No bootstrap required, opened a new log
I20260929 11:31:55.080582 11660 raft_consensus.cc:348] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a994fba1c5a84a149b8f98f636d29294" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34091 } }
I20260929 11:31:55.081032 11660 raft_consensus.cc:374] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:55.081142 11660 raft_consensus.cc:729] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a994fba1c5a84a149b8f98f636d29294, State: Initialized, Role: FOLLOWER
I20260929 11:31:55.081772 11660 consensus_queue.cc:260] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a994fba1c5a84a149b8f98f636d29294" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34091 } }
I20260929 11:31:55.082029 11660 raft_consensus.cc:388] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:31:55.082178 11660 raft_consensus.cc:482] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:31:55.082347 11660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:55.084757 11660 raft_consensus.cc:504] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a994fba1c5a84a149b8f98f636d29294" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34091 } }
I20260929 11:31:55.085388 11660 leader_election.cc:302] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a994fba1c5a84a149b8f98f636d29294; no voters:
I20260929 11:31:55.086226 11660 leader_election.cc:288] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:31:55.086266 11665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:31:55.087291 11665 raft_consensus.cc:686] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294 [term 1 LEADER]: Becoming Leader. State: Replica: a994fba1c5a84a149b8f98f636d29294, State: Running, Role: LEADER
I20260929 11:31:55.087859 11665 consensus_queue.cc:237] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a994fba1c5a84a149b8f98f636d29294" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34091 } }
I20260929 11:31:55.088490 11660 sys_catalog.cc:564] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:31:55.090828 11667 sys_catalog.cc:455] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a994fba1c5a84a149b8f98f636d29294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a994fba1c5a84a149b8f98f636d29294" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34091 } } }
I20260929 11:31:55.091370 11667 sys_catalog.cc:458] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294 [sys.catalog]: This master's current role is: LEADER
I20260929 11:31:55.091420 11666 sys_catalog.cc:455] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a994fba1c5a84a149b8f98f636d29294. Latest consensus state: current_term: 1 leader_uuid: "a994fba1c5a84a149b8f98f636d29294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a994fba1c5a84a149b8f98f636d29294" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34091 } } }
I20260929 11:31:55.091732 11666 sys_catalog.cc:458] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294 [sys.catalog]: This master's current role is: LEADER
I20260929 11:31:55.092439 11674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:31:55.098412 11674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:31:55.110574 11674 catalog_manager.cc:1347] Generated new cluster ID: ff2fbba06cbb4934a9e1177aa28b2be9
I20260929 11:31:55.110707 11674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:31:55.126683 11674 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:31:55.128969 11674 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:31:55.136399 11674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294: Generated new TSK 0
I20260929 11:31:55.137761 11674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:31:55.154227 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.1:0
--local_ip_for_outbound_sockets=127.23.153.1
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:34091
--builtin_ntp_servers=127.23.153.20:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 11:31:55.557299 11684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:55.557600 11684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:55.557708 11684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:55.565896 11684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:55.566047 11684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:55.566208 11684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.1
I20260929 11:31:55.576139 11684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.153.1
--webserver_port=0
--tserver_master_addrs=127.23.153.62:34091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.1
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:55.577481 11684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:55.579505 11684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:55.593685 11692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:55.593827 11689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:55.593775 11690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:55.594966 11684 server_base.cc:1034] running on GCE node
I20260929 11:31:55.595763 11684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:55.596942 11684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:55.598176 11684 hybrid_clock.cc:648] HybridClock initialized: now 1790681515598113 us; error 62 us; skew 500 ppm
I20260929 11:31:55.598539 11684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:31:55.601606 11684 webserver.cc:460] Webserver started at http://127.23.153.1:42757/ using document root <none> and password file <none>
I20260929 11:31:55.602409 11684 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:55.602525 11684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:55.602854 11684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:55.606170 11684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-0/data/instance:
uuid: "57401538bfbd45b2bc2a90f5b8b6fbfb"
format_stamp: "Formatted at 2026-09-29 11:31:55 on dist-test-slave-qk05"
server_key: "7f1b770925cbd298770198ac71293dec"
server_key_iv: "6ab62e0de2b7baa7b5eb15a2ff014db4"
server_key_version: "encryptionkey@0"
I20260929 11:31:55.607725 11684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance:
uuid: "57401538bfbd45b2bc2a90f5b8b6fbfb"
format_stamp: "Formatted at 2026-09-29 11:31:55 on dist-test-slave-qk05"
server_key: "7f1b770925cbd298770198ac71293dec"
server_key_iv: "6ab62e0de2b7baa7b5eb15a2ff014db4"
server_key_version: "encryptionkey@0"
I20260929 11:31:55.614091 11684 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:31:55.618276 11698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:55.619889 11684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:31:55.620082 11684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-0/wal
uuid: "57401538bfbd45b2bc2a90f5b8b6fbfb"
format_stamp: "Formatted at 2026-09-29 11:31:55 on dist-test-slave-qk05"
server_key: "7f1b770925cbd298770198ac71293dec"
server_key_iv: "6ab62e0de2b7baa7b5eb15a2ff014db4"
server_key_version: "encryptionkey@0"
I20260929 11:31:55.620322 11684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:55.661207 11684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:55.662145 11684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:55.662460 11684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:55.663558 11684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:55.665186 11684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:55.665293 11684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:55.665400 11684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:55.665477 11684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:55.697329 11684 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.1:39405
I20260929 11:31:55.697371 11810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.1:39405 every 8 connection(s)
I20260929 11:31:55.698853 11684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-0/data/info.pb
I20260929 11:31:55.707475 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 11684
I20260929 11:31:55.707756 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-0/wal/instance
I20260929 11:31:55.708348 24164 external_mini_cluster.cc:1428] Setting key 55315d230fe1f8b25d2bb2865b0317c6
I20260929 11:31:55.712244 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.2:0
--local_ip_for_outbound_sockets=127.23.153.2
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:34091
--builtin_ntp_servers=127.23.153.20:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 11:31:55.717219 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:55.703340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.1:52881 (local address 127.23.153.62:34091)
0929 11:31:55.705246 (+ 1906us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:55.705250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:55.705271 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:31:55.706632 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:55.706639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:55.706778 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:55.706931 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:55.710895 (+ 3964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:55.711947 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:55.713916 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:55.714527 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:55.714671 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:55.716284 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:55.716291 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:55.716296 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:55.716322 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:55.716373 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:55.716390 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:55.716499 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:55.716734 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 11:31:55.716853 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1790,"thread_start_us":1657,"threads_started":1}
I20260929 11:31:55.718324 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:55.702561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:34091 (local address 127.23.153.1:52881)
0929 11:31:55.703832 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:55.703890 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:55.706228 (+ 2338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:55.709431 (+ 3203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:55.709450 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:55.709644 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:31:55.710633 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:55.710666 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:55.712193 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:55.712198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:55.713690 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:55.713709 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:55.714799 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:55.714806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:55.715099 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:55.715856 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:31:55.715858 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:31:55.715910 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:31:55.716024 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:31:55.716501 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:31:55.716508 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:31:55.716515 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 11:31:55.716674 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 11:31:55.717048 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":395,"threads_started":1}
I20260929 11:31:55.720628 11811 heartbeater.cc:347] Connected to a master server at 127.23.153.62:34091
I20260929 11:31:55.721169 11811 heartbeater.cc:464] Registering TS with master...
I20260929 11:31:55.722239 11811 heartbeater.cc:511] Master 127.23.153.62:34091 requested a full tablet report, sending...
I20260929 11:31:55.724963 11625 ts_manager.cc:194] Registered new tserver with Master: 57401538bfbd45b2bc2a90f5b8b6fbfb (127.23.153.1:39405)
I20260929 11:31:55.726643 11625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.153.1:52881
W20260929 11:31:56.070377 11815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:56.070827 11815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:56.070994 11815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:56.079720 11815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:56.079843 11815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:56.080004 11815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.2
I20260929 11:31:56.090207 11815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.153.2
--webserver_port=0
--tserver_master_addrs=127.23.153.62:34091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.2
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:56.091651 11815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:56.093540 11815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:56.107558 11820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:56.107542 11821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:56.108279 11823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:56.108635 11815 server_base.cc:1034] running on GCE node
I20260929 11:31:56.109314 11815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:56.110486 11815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:56.111716 11815 hybrid_clock.cc:648] HybridClock initialized: now 1790681516111651 us; error 70 us; skew 500 ppm
I20260929 11:31:56.112084 11815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:31:56.114562 11815 webserver.cc:460] Webserver started at http://127.23.153.2:37383/ using document root <none> and password file <none>
I20260929 11:31:56.115376 11815 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:56.115496 11815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:56.115823 11815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:56.119232 11815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-1/data/instance:
uuid: "0a709be3f24a43489f3744c96ce985b6"
format_stamp: "Formatted at 2026-09-29 11:31:56 on dist-test-slave-qk05"
server_key: "dbbaf6e6599af86fa89e5c9bd80357bd"
server_key_iv: "861947c61dc21e8af0187ab4fd6f78f9"
server_key_version: "encryptionkey@0"
I20260929 11:31:56.120177 11815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance:
uuid: "0a709be3f24a43489f3744c96ce985b6"
format_stamp: "Formatted at 2026-09-29 11:31:56 on dist-test-slave-qk05"
server_key: "dbbaf6e6599af86fa89e5c9bd80357bd"
server_key_iv: "861947c61dc21e8af0187ab4fd6f78f9"
server_key_version: "encryptionkey@0"
I20260929 11:31:56.126538 11815 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 11:31:56.131300 11829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:56.133071 11815 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:31:56.133285 11815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-1/wal
uuid: "0a709be3f24a43489f3744c96ce985b6"
format_stamp: "Formatted at 2026-09-29 11:31:56 on dist-test-slave-qk05"
server_key: "dbbaf6e6599af86fa89e5c9bd80357bd"
server_key_iv: "861947c61dc21e8af0187ab4fd6f78f9"
server_key_version: "encryptionkey@0"
I20260929 11:31:56.133553 11815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:56.157419 11815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:56.158339 11815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:56.158784 11815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:56.159960 11815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:56.161650 11815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:56.161772 11815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:56.161887 11815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:56.161962 11815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:56.194864 11815 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.2:39117
I20260929 11:31:56.194944 11941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.2:39117 every 8 connection(s)
I20260929 11:31:56.196408 11815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-1/data/info.pb
I20260929 11:31:56.202720 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 11815
I20260929 11:31:56.202956 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-1/wal/instance
I20260929 11:31:56.203498 24164 external_mini_cluster.cc:1428] Setting key f190dccc73b0d24582b476b1f2297d97
I20260929 11:31:56.206796 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.153.3:0
--local_ip_for_outbound_sockets=127.23.153.3
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:34091
--builtin_ntp_servers=127.23.153.20:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 11:31:56.213019 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:56.199623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:39813 (local address 127.23.153.62:34091)
0929 11:31:56.199813 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:56.199819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:56.202019 (+ 2200us) server_negotiation.cc:407] Connection header received
0929 11:31:56.204608 (+ 2589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:56.204613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:56.204717 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:56.204839 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:56.206314 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.207152 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.209770 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.210321 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.210445 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:56.212197 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:56.212204 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:56.212208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:56.212232 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:56.212269 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:56.212284 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:56.212365 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:56.212614 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 11:31:56.212716 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:31:56.214758 11944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:56.200381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:34091 (local address 127.23.153.2:39813)
0929 11:31:56.201835 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:56.201879 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:56.204167 (+ 2288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:56.205077 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:56.205087 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:56.205202 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:31:56.206111 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:56.206143 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.207328 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.207335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:56.209536 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:56.209560 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.210599 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.210606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:56.210915 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:56.211772 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:31:56.211775 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:31:56.211836 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:31:56.211959 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:31:56.212380 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:31:56.212389 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:31:56.212397 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 11:31:56.212609 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 11:31:56.213169 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":531,"threads_started":1}
I20260929 11:31:56.216511 11942 heartbeater.cc:347] Connected to a master server at 127.23.153.62:34091
I20260929 11:31:56.216902 11942 heartbeater.cc:464] Registering TS with master...
I20260929 11:31:56.217821 11942 heartbeater.cc:511] Master 127.23.153.62:34091 requested a full tablet report, sending...
I20260929 11:31:56.219769 11625 ts_manager.cc:194] Registered new tserver with Master: 0a709be3f24a43489f3744c96ce985b6 (127.23.153.2:39117)
I20260929 11:31:56.220857 11625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.153.2:39813
W20260929 11:31:56.580444 11945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:56.580763 11945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:56.580854 11945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:56.589111 11945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:56.589247 11945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:56.589381 11945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.153.3
I20260929 11:31:56.599895 11945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.153.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.153.3
--webserver_port=0
--tserver_master_addrs=127.23.153.62:34091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.153.3
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:56.601413 11945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:56.603439 11945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:56.619868 11951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:56.619918 11953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:56.620114 11950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:56.620715 11945 server_base.cc:1034] running on GCE node
I20260929 11:31:56.621372 11945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:56.622447 11945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:56.623690 11945 hybrid_clock.cc:648] HybridClock initialized: now 1790681516623630 us; error 48 us; skew 500 ppm
I20260929 11:31:56.624065 11945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 11:31:56.626718 11945 webserver.cc:460] Webserver started at http://127.23.153.3:36455/ using document root <none> and password file <none>
I20260929 11:31:56.627594 11945 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:56.627700 11945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:56.628024 11945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:56.631238 11945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-2/data/instance:
uuid: "16b330a5be9e4e7894498e72362e6594"
format_stamp: "Formatted at 2026-09-29 11:31:56 on dist-test-slave-qk05"
server_key: "f18084fb39f502fe04fadde640f9faf8"
server_key_iv: "6d740392edb119fbe16f8136c8567702"
server_key_version: "encryptionkey@0"
I20260929 11:31:56.632148 11945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance:
uuid: "16b330a5be9e4e7894498e72362e6594"
format_stamp: "Formatted at 2026-09-29 11:31:56 on dist-test-slave-qk05"
server_key: "f18084fb39f502fe04fadde640f9faf8"
server_key_iv: "6d740392edb119fbe16f8136c8567702"
server_key_version: "encryptionkey@0"
I20260929 11:31:56.638659 11945 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 11:31:56.643148 11959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:56.645052 11945 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260929 11:31:56.645267 11945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-2/wal
uuid: "16b330a5be9e4e7894498e72362e6594"
format_stamp: "Formatted at 2026-09-29 11:31:56 on dist-test-slave-qk05"
server_key: "f18084fb39f502fe04fadde640f9faf8"
server_key_iv: "6d740392edb119fbe16f8136c8567702"
server_key_version: "encryptionkey@0"
I20260929 11:31:56.645531 11945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:56.677809 11945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:56.678736 11945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:56.679072 11945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:56.680150 11945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 11:31:56.681835 11945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 11:31:56.681950 11945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:56.682072 11945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 11:31:56.682145 11945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:56.716089 11945 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.3:39249
I20260929 11:31:56.716156 12071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.3:39249 every 8 connection(s)
I20260929 11:31:56.717463 11945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-2/data/info.pb
I20260929 11:31:56.718102 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 11945
I20260929 11:31:56.718261 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-2/wal/instance
I20260929 11:31:56.718851 24164 external_mini_cluster.cc:1428] Setting key dbaaaed113df28d42ed0f7cc6ad3d0d2
I20260929 11:31:56.731531 11811 heartbeater.cc:503] Master 127.23.153.62:34091 was elected leader, sending a full tablet report...
I20260929 11:31:56.731954 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:56.720146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.3:50293 (local address 127.23.153.62:34091)
0929 11:31:56.720653 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:56.720659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:56.722182 (+ 1523us) server_negotiation.cc:407] Connection header received
0929 11:31:56.724203 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:56.724208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:56.724290 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:56.724394 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:56.725656 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.726493 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.728843 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.729284 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.729399 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:56.730825 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:56.730831 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:56.730834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:56.730857 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:56.730901 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:56.730914 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:56.730992 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:56.731249 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 11:31:56.731344 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":245,"threads_started":1}
I20260929 11:31:56.733107 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:56.720933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:34091 (local address 127.23.153.3:50293)
0929 11:31:56.722027 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:56.722072 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:56.723817 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:56.724645 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:56.724655 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:56.724757 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:31:56.725432 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:56.725463 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.726683 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.726688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:56.728645 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:56.728664 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.729523 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.729528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:56.729758 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:56.730480 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 11:31:56.730482 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:31:56.730528 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 11:31:56.730622 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:31:56.730997 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:31:56.731003 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:31:56.731008 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 11:31:56.731216 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 11:31:56.731699 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":325,"threads_started":1}
I20260929 11:31:56.734730 12072 heartbeater.cc:347] Connected to a master server at 127.23.153.62:34091
I20260929 11:31:56.735085 12072 heartbeater.cc:464] Registering TS with master...
I20260929 11:31:56.735960 12072 heartbeater.cc:511] Master 127.23.153.62:34091 requested a full tablet report, sending...
I20260929 11:31:56.737792 11624 ts_manager.cc:194] Registered new tserver with Master: 16b330a5be9e4e7894498e72362e6594 (127.23.153.3:39249)
I20260929 11:31:56.738754 11624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.153.3:50293
I20260929 11:31:56.748061 24164 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 11:31:56.761245 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:56.754544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.23.153.62:34091)
0929 11:31:56.754757 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:56.754763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:56.755426 (+ 663us) server_negotiation.cc:407] Connection header received
0929 11:31:56.755608 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:56.755613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:56.755699 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:56.755807 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:56.756293 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.757051 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.758746 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.759318 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.759457 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:56.760570 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:56.760576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:56.760580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:56.760604 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:56.760648 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:56.760662 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:56.760741 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:56.760905 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 11:31:56.760998 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 11:31:56.766243 11624 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 11:31:56.773380 11624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 11:31:56.801846 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:56.786929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:39249 (local address 127.0.0.1:51784)
0929 11:31:56.788253 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:56.788309 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:56.788591 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:56.794206 (+ 5615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:56.794211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:56.794238 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:56.794824 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:56.794838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.796555 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.796561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:56.798445 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:56.798464 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.800824 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.800830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:56.801064 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:56.801132 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 11:31:56.801235 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 11:31:56.801466 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":47,"thread_start_us":1869,"threads_started":1}
I20260929 11:31:56.802176 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:56.788740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39405 (local address 127.0.0.1:41686)
0929 11:31:56.792028 (+ 3288us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:56.792051 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:56.792197 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:56.793933 (+ 1736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:56.793937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:56.793966 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:56.794787 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:56.794804 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.797067 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.797075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:56.799496 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:56.799512 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.801411 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.801417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:56.801632 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:56.801664 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 11:31:56.801754 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 11:31:56.801892 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3137,"thread_start_us":997,"threads_started":1}
I20260929 11:31:56.802234 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:56.787766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.23.153.2:39117)
0929 11:31:56.792791 (+ 5025us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:56.792803 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:56.792843 (+ 40us) server_negotiation.cc:407] Connection header received
0929 11:31:56.792977 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:56.792989 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:56.793322 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:56.793633 (+ 311us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:56.794375 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.795890 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.799196 (+ 3306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.800734 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.801405 (+ 671us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:56.801626 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:56.801704 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 11:31:56.801893 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4819,"thread_start_us":4675,"threads_started":1}
I20260929 11:31:56.802677 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:56.787213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.2:39117 (local address 127.0.0.1:46462)
0929 11:31:56.789073 (+ 1860us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:56.789094 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:56.789236 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:56.793668 (+ 4432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:56.793675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:56.793710 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:56.794148 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:56.794163 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.796072 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.796078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:56.798874 (+ 2796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:56.798892 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.800976 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.800983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:56.801187 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:56.801221 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 11:31:56.801279 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 11:31:56.802137 (+ 858us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1739,"mutex_wait_us":728,"thread_start_us":561,"threads_started":1}
I20260929 11:31:56.802713 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:56.789383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41686 (local address 127.23.153.1:39405)
0929 11:31:56.793225 (+ 3842us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:56.793234 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:56.793268 (+ 34us) server_negotiation.cc:407] Connection header received
0929 11:31:56.793387 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:56.793394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:56.793640 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:56.793843 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:56.795463 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.796892 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.799747 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.801200 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.801334 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:56.801631 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:56.801804 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 11:31:56.801955 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3657,"thread_start_us":3497,"threads_started":1}
I20260929 11:31:56.804263 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:56.787630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.23.153.3:39249)
0929 11:31:56.793602 (+ 5972us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:56.793610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:56.793643 (+ 33us) server_negotiation.cc:407] Connection header received
0929 11:31:56.793738 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:56.793746 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:56.793982 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:56.794185 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:56.795071 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.796390 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.798768 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.800476 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.803094 (+ 2618us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:56.803675 (+ 581us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:56.803759 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 11:31:56.803948 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5793,"thread_start_us":5635,"threads_started":1}
I20260929 11:31:56.806944 11746 tablet_service.cc:1467] Processing CreateTablet for tablet 498819b5af464eb5b3f1bbef6cb56b31 (DEFAULT_TABLE table=test-table [id=f7ea71e020c04e1c88fa96ab1bbe8c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:31:56.806948 11877 tablet_service.cc:1467] Processing CreateTablet for tablet 498819b5af464eb5b3f1bbef6cb56b31 (DEFAULT_TABLE table=test-table [id=f7ea71e020c04e1c88fa96ab1bbe8c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:31:56.808656 11877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 498819b5af464eb5b3f1bbef6cb56b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:56.808660 11746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 498819b5af464eb5b3f1bbef6cb56b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:56.808717 12007 tablet_service.cc:1467] Processing CreateTablet for tablet 498819b5af464eb5b3f1bbef6cb56b31 (DEFAULT_TABLE table=test-table [id=f7ea71e020c04e1c88fa96ab1bbe8c17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 11:31:56.810097 12007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 498819b5af464eb5b3f1bbef6cb56b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:56.823643 12091 tablet_bootstrap.cc:492] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594: Bootstrap starting.
I20260929 11:31:56.826256 12093 tablet_bootstrap.cc:492] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6: Bootstrap starting.
I20260929 11:31:56.828516 12091 tablet_bootstrap.cc:654] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:56.830272 12091 log.cc:824] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:56.830437 12093 tablet_bootstrap.cc:654] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:56.830818 12092 tablet_bootstrap.cc:492] T 498819b5af464eb5b3f1bbef6cb56b31 P 57401538bfbd45b2bc2a90f5b8b6fbfb: Bootstrap starting.
I20260929 11:31:56.831825 12093 log.cc:824] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:56.834218 12091 tablet_bootstrap.cc:492] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594: No bootstrap required, opened a new log
I20260929 11:31:56.834542 12091 ts_tablet_manager.cc:1397] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260929 11:31:56.835697 12092 tablet_bootstrap.cc:654] T 498819b5af464eb5b3f1bbef6cb56b31 P 57401538bfbd45b2bc2a90f5b8b6fbfb: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:56.837458 12092 log.cc:824] T 498819b5af464eb5b3f1bbef6cb56b31 P 57401538bfbd45b2bc2a90f5b8b6fbfb: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:56.837940 12093 tablet_bootstrap.cc:492] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6: No bootstrap required, opened a new log
I20260929 11:31:56.838281 12093 ts_tablet_manager.cc:1397] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 11:31:56.842960 12092 tablet_bootstrap.cc:492] T 498819b5af464eb5b3f1bbef6cb56b31 P 57401538bfbd45b2bc2a90f5b8b6fbfb: No bootstrap required, opened a new log
I20260929 11:31:56.843346 12092 ts_tablet_manager.cc:1397] T 498819b5af464eb5b3f1bbef6cb56b31 P 57401538bfbd45b2bc2a90f5b8b6fbfb: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260929 11:31:56.843626 12091 raft_consensus.cc:348] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b330a5be9e4e7894498e72362e6594" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39249 } } peers { permanent_uuid: "0a709be3f24a43489f3744c96ce985b6" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39117 } } peers { permanent_uuid: "57401538bfbd45b2bc2a90f5b8b6fbfb" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39405 } }
I20260929 11:31:56.844053 12091 raft_consensus.cc:374] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:56.844163 12091 raft_consensus.cc:729] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b330a5be9e4e7894498e72362e6594, State: Initialized, Role: FOLLOWER
I20260929 11:31:56.844805 12091 consensus_queue.cc:260] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b330a5be9e4e7894498e72362e6594" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39249 } } peers { permanent_uuid: "0a709be3f24a43489f3744c96ce985b6" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39117 } } peers { permanent_uuid: "57401538bfbd45b2bc2a90f5b8b6fbfb" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39405 } }
I20260929 11:31:56.846036 12072 heartbeater.cc:503] Master 127.23.153.62:34091 was elected leader, sending a full tablet report...
I20260929 11:31:56.846606 12091 ts_tablet_manager.cc:1428] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260929 11:31:56.848691 12093 raft_consensus.cc:348] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b330a5be9e4e7894498e72362e6594" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39249 } } peers { permanent_uuid: "0a709be3f24a43489f3744c96ce985b6" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39117 } } peers { permanent_uuid: "57401538bfbd45b2bc2a90f5b8b6fbfb" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39405 } }
I20260929 11:31:56.849102 12093 raft_consensus.cc:374] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:56.849210 12093 raft_consensus.cc:729] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a709be3f24a43489f3744c96ce985b6, State: Initialized, Role: FOLLOWER
I20260929 11:31:56.849941 12093 consensus_queue.cc:260] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b330a5be9e4e7894498e72362e6594" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39249 } } peers { permanent_uuid: "0a709be3f24a43489f3744c96ce985b6" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39117 } } peers { permanent_uuid: "57401538bfbd45b2bc2a90f5b8b6fbfb" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39405 } }
I20260929 11:31:56.852717 11942 heartbeater.cc:503] Master 127.23.153.62:34091 was elected leader, sending a full tablet report...
I20260929 11:31:56.853530 12093 ts_tablet_manager.cc:1428] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6: Time spent starting tablet: real 0.015s user 0.014s sys 0.003s
I20260929 11:31:56.854562 12092 raft_consensus.cc:348] T 498819b5af464eb5b3f1bbef6cb56b31 P 57401538bfbd45b2bc2a90f5b8b6fbfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b330a5be9e4e7894498e72362e6594" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39249 } } peers { permanent_uuid: "0a709be3f24a43489f3744c96ce985b6" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39117 } } peers { permanent_uuid: "57401538bfbd45b2bc2a90f5b8b6fbfb" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39405 } }
I20260929 11:31:56.855127 12092 raft_consensus.cc:374] T 498819b5af464eb5b3f1bbef6cb56b31 P 57401538bfbd45b2bc2a90f5b8b6fbfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:56.855253 12092 raft_consensus.cc:729] T 498819b5af464eb5b3f1bbef6cb56b31 P 57401538bfbd45b2bc2a90f5b8b6fbfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57401538bfbd45b2bc2a90f5b8b6fbfb, State: Initialized, Role: FOLLOWER
I20260929 11:31:56.855495 12098 raft_consensus.cc:482] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 11:31:56.856070 12092 consensus_queue.cc:260] T 498819b5af464eb5b3f1bbef6cb56b31 P 57401538bfbd45b2bc2a90f5b8b6fbfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b330a5be9e4e7894498e72362e6594" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39249 } } peers { permanent_uuid: "0a709be3f24a43489f3744c96ce985b6" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39117 } } peers { permanent_uuid: "57401538bfbd45b2bc2a90f5b8b6fbfb" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39405 } }
I20260929 11:31:56.855825 12098 raft_consensus.cc:504] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b330a5be9e4e7894498e72362e6594" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39249 } } peers { permanent_uuid: "0a709be3f24a43489f3744c96ce985b6" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39117 } } peers { permanent_uuid: "57401538bfbd45b2bc2a90f5b8b6fbfb" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39405 } }
I20260929 11:31:56.858033 12092 ts_tablet_manager.cc:1428] T 498819b5af464eb5b3f1bbef6cb56b31 P 57401538bfbd45b2bc2a90f5b8b6fbfb: Time spent starting tablet: real 0.015s user 0.011s sys 0.001s
I20260929 11:31:56.860250 12098 leader_election.cc:288] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16b330a5be9e4e7894498e72362e6594 (127.23.153.3:39249), 57401538bfbd45b2bc2a90f5b8b6fbfb (127.23.153.1:39405)
I20260929 11:31:56.867401 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:56.858871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.3:39249 (local address 127.23.153.2:50699)
0929 11:31:56.860215 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:56.860239 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:56.860751 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:56.861351 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:56.861356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:56.861387 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:56.861763 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:56.861775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.863134 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.863138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:56.865049 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:56.865069 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.866690 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.866695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:56.866824 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:56.866853 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 11:31:56.866945 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 11:31:56.867118 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1190,"thread_start_us":974,"threads_started":1}
I20260929 11:31:56.867426 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:56.859084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:50699 (local address 127.23.153.3:39249)
0929 11:31:56.859296 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:56.859302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:56.860329 (+ 1027us) server_negotiation.cc:407] Connection header received
0929 11:31:56.860966 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:56.860974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:56.861083 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:56.861148 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:56.861986 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.862991 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.865336 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.866443 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.866639 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:56.866714 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:56.866986 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 11:31:56.867180 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 11:31:56.868333 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:56.860670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.153.2:55659 (local address 127.23.153.1:39405)
0929 11:31:56.860871 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:56.860877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:56.861700 (+ 823us) server_negotiation.cc:407] Connection header received
0929 11:31:56.861892 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:56.861898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:56.861985 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:56.862064 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 11:31:56.862619 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.863727 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.865991 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.867249 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.867431 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:56.867500 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:56.868007 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 11:31:56.868116 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:31:56.868358 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:56.860624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39405 (local address 127.23.153.2:55659)
0929 11:31:56.861574 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:56.861596 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:31:56.861745 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:31:56.862095 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:31:56.862100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:31:56.862124 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 11:31:56.862452 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:56.862465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.863946 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.863952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:56.865732 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:31:56.865746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.867498 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.867504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:31:56.867810 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:56.867845 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 11:31:56.867960 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 11:31:56.868095 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"thread_start_us":580,"threads_started":1}
I20260929 11:31:56.868793 12027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498819b5af464eb5b3f1bbef6cb56b31" candidate_uuid: "0a709be3f24a43489f3744c96ce985b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b330a5be9e4e7894498e72362e6594" is_pre_election: true
I20260929 11:31:56.869354 12027 raft_consensus.cc:2454] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a709be3f24a43489f3744c96ce985b6 in term 0.
I20260929 11:31:56.869311 11766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498819b5af464eb5b3f1bbef6cb56b31" candidate_uuid: "0a709be3f24a43489f3744c96ce985b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57401538bfbd45b2bc2a90f5b8b6fbfb" is_pre_election: true
I20260929 11:31:56.869740 11766 raft_consensus.cc:2454] T 498819b5af464eb5b3f1bbef6cb56b31 P 57401538bfbd45b2bc2a90f5b8b6fbfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a709be3f24a43489f3744c96ce985b6 in term 0.
I20260929 11:31:56.870090 11833 leader_election.cc:302] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a709be3f24a43489f3744c96ce985b6, 16b330a5be9e4e7894498e72362e6594; no voters:
I20260929 11:31:56.870568 12098 raft_consensus.cc:2781] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 11:31:56.870741 12098 raft_consensus.cc:482] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 11:31:56.870848 12098 raft_consensus.cc:3037] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:56.873335 12098 raft_consensus.cc:504] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b330a5be9e4e7894498e72362e6594" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39249 } } peers { permanent_uuid: "0a709be3f24a43489f3744c96ce985b6" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39117 } } peers { permanent_uuid: "57401538bfbd45b2bc2a90f5b8b6fbfb" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39405 } }
I20260929 11:31:56.873980 12098 leader_election.cc:288] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [CANDIDATE]: Term 1 election: Requested vote from peers 16b330a5be9e4e7894498e72362e6594 (127.23.153.3:39249), 57401538bfbd45b2bc2a90f5b8b6fbfb (127.23.153.1:39405)
I20260929 11:31:56.874630 11766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498819b5af464eb5b3f1bbef6cb56b31" candidate_uuid: "0a709be3f24a43489f3744c96ce985b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57401538bfbd45b2bc2a90f5b8b6fbfb"
I20260929 11:31:56.874670 12027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "498819b5af464eb5b3f1bbef6cb56b31" candidate_uuid: "0a709be3f24a43489f3744c96ce985b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b330a5be9e4e7894498e72362e6594"
I20260929 11:31:56.874848 11766 raft_consensus.cc:3037] T 498819b5af464eb5b3f1bbef6cb56b31 P 57401538bfbd45b2bc2a90f5b8b6fbfb [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:56.874893 12027 raft_consensus.cc:3037] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:56.877274 11766 raft_consensus.cc:2454] T 498819b5af464eb5b3f1bbef6cb56b31 P 57401538bfbd45b2bc2a90f5b8b6fbfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a709be3f24a43489f3744c96ce985b6 in term 1.
I20260929 11:31:56.877836 11832 leader_election.cc:302] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a709be3f24a43489f3744c96ce985b6, 57401538bfbd45b2bc2a90f5b8b6fbfb; no voters:
I20260929 11:31:56.878031 12027 raft_consensus.cc:2454] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a709be3f24a43489f3744c96ce985b6 in term 1.
I20260929 11:31:56.878221 12098 raft_consensus.cc:2781] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:31:56.878726 12098 raft_consensus.cc:686] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [term 1 LEADER]: Becoming Leader. State: Replica: 0a709be3f24a43489f3744c96ce985b6, State: Running, Role: LEADER
I20260929 11:31:56.879314 12098 consensus_queue.cc:237] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b330a5be9e4e7894498e72362e6594" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39249 } } peers { permanent_uuid: "0a709be3f24a43489f3744c96ce985b6" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39117 } } peers { permanent_uuid: "57401538bfbd45b2bc2a90f5b8b6fbfb" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39405 } }
I20260929 11:31:56.884182 11625 catalog_manager.cc:5495] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a709be3f24a43489f3744c96ce985b6 (127.23.153.2). New cstate: current_term: 1 leader_uuid: "0a709be3f24a43489f3744c96ce985b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16b330a5be9e4e7894498e72362e6594" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a709be3f24a43489f3744c96ce985b6" member_type: VOTER last_known_addr { host: "127.23.153.2" port: 39117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57401538bfbd45b2bc2a90f5b8b6fbfb" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39405 } health_report { overall_health: UNKNOWN } } }
I20260929 11:31:56.904385 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:56.898896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46468 (local address 127.23.153.2:39117)
0929 11:31:56.899142 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:56.899147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:56.899165 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:31:56.899259 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:56.899263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:56.899360 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:56.899500 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:31:56.899858 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.900560 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.902054 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.902497 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.902606 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:56.903171 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:31:56.903733 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:31:56.903850 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:56.904006 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 11:31:56.904107 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 11:31:56.913764 12027 raft_consensus.cc:1261] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594 [term 1 FOLLOWER]: Refusing update from remote peer 0a709be3f24a43489f3744c96ce985b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:31:56.913836 11766 raft_consensus.cc:1261] T 498819b5af464eb5b3f1bbef6cb56b31 P 57401538bfbd45b2bc2a90f5b8b6fbfb [term 1 FOLLOWER]: Refusing update from remote peer 0a709be3f24a43489f3744c96ce985b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:31:56.914912 12098 consensus_queue.cc:1035] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57401538bfbd45b2bc2a90f5b8b6fbfb" member_type: VOTER last_known_addr { host: "127.23.153.1" port: 39405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:31:56.915307 12102 consensus_queue.cc:1035] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b330a5be9e4e7894498e72362e6594" member_type: VOTER last_known_addr { host: "127.23.153.3" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:31:56.951426 11624 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:31:56.951771 11624 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 11:31:56.957039 11624 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a994fba1c5a84a149b8f98f636d29294: Sending DeleteTablet for 3 replicas of tablet 498819b5af464eb5b3f1bbef6cb56b31
I20260929 11:31:56.958496 12007 tablet_service.cc:1514] Processing DeleteTablet for tablet 498819b5af464eb5b3f1bbef6cb56b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:56 UTC) from {username='slave'} at 127.0.0.1:51784
I20260929 11:31:56.959322 11746 tablet_service.cc:1514] Processing DeleteTablet for tablet 498819b5af464eb5b3f1bbef6cb56b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:56 UTC) from {username='slave'} at 127.0.0.1:41686
I20260929 11:31:56.960901 11877 tablet_service.cc:1514] Processing DeleteTablet for tablet 498819b5af464eb5b3f1bbef6cb56b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:56 UTC) from {username='slave'} at 127.0.0.1:46462
I20260929 11:31:56.962186 11745 tablet_service.cc:1514] Processing DeleteTablet for tablet 498819b5af464eb5b3f1bbef6cb56b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 11:31:56 UTC) from {username='slave'} at 127.0.0.1:41686
I20260929 11:31:56.963270 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 11684
I20260929 11:31:56.971581 12115 tablet_replica.cc:330] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594: stopping tablet replica
I20260929 11:31:56.972039 12115 raft_consensus.cc:2229] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 11:31:56.972556 12115 raft_consensus.cc:2258] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:31:56.978899 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:56.965837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41688 (local address 127.23.153.1:39405)
0929 11:31:56.966041 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:56.966048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:56.968148 (+ 2100us) server_negotiation.cc:407] Connection header received
0929 11:31:56.969062 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:56.969070 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:56.969191 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:56.969271 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:56.970446 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.971806 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.974075 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:56.974807 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:56.974994 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:56.976772 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:56.976787 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:56.976808 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:56.977783 (+ 975us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:56.977859 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:56.977891 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:56.978021 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:56.978339 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 11:31:56.978491 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 11:31:56.978915 12117 tablet_replica.cc:330] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6: stopping tablet replica
I20260929 11:31:56.980656 12117 raft_consensus.cc:2229] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 11:31:56.980878 11806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:56.981796 12117 raft_consensus.cc:2258] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 11:31:56.983820 12115 ts_tablet_manager.cc:1905] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:31:56.984316 12117 ts_tablet_manager.cc:1905] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 11:31:56.990666 12115 ts_tablet_manager.cc:1918] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:31:56.990880 12115 log.cc:1196] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-2/wal/wals/498819b5af464eb5b3f1bbef6cb56b31
I20260929 11:31:56.991458 12115 ts_tablet_manager.cc:1939] T 498819b5af464eb5b3f1bbef6cb56b31 P 16b330a5be9e4e7894498e72362e6594: Deleting consensus metadata
I20260929 11:31:56.992448 12117 ts_tablet_manager.cc:1918] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 11:31:56.992650 12117 log.cc:1196] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6: Deleting WAL directory at /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790681181154781-24164-0/minicluster-data/ts-1/wal/wals/498819b5af464eb5b3f1bbef6cb56b31
I20260929 11:31:56.993201 12117 ts_tablet_manager.cc:1939] T 498819b5af464eb5b3f1bbef6cb56b31 P 0a709be3f24a43489f3744c96ce985b6: Deleting consensus metadata
I20260929 11:31:56.993654 11612 catalog_manager.cc:4841] TS 16b330a5be9e4e7894498e72362e6594 (127.23.153.3:39249): tablet 498819b5af464eb5b3f1bbef6cb56b31 (table test-table [id=f7ea71e020c04e1c88fa96ab1bbe8c17]) successfully deleted
I20260929 11:31:56.995335 11611 catalog_manager.cc:4841] TS 0a709be3f24a43489f3744c96ce985b6 (127.23.153.2:39117): tablet 498819b5af464eb5b3f1bbef6cb56b31 (table test-table [id=f7ea71e020c04e1c88fa96ab1bbe8c17]) successfully deleted
I20260929 11:31:57.093160 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 11684
I20260929 11:31:57.093344 12116 tablet_replica.cc:330] T 498819b5af464eb5b3f1bbef6cb56b31 P 57401538bfbd45b2bc2a90f5b8b6fbfb: stopping tablet replica
I20260929 11:31:57.108729 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 11815
W20260929 11:31:57.108947 11611 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.153.1:39405 (error 108)
I20260929 11:31:57.111104 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:57.110552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39405 (local address 127.0.0.1:41702)
0929 11:31:57.110725 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:57.110884 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 11:31:57.111465 11611 catalog_manager.cc:4584] TS 57401538bfbd45b2bc2a90f5b8b6fbfb (127.23.153.1:39405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 498819b5af464eb5b3f1bbef6cb56b31: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39405: connect: Connection refused (error 111)
I20260929 11:31:57.117369 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:57.109589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.23.153.2:39117)
0929 11:31:57.109775 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:57.109781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:57.109804 (+ 23us) server_negotiation.cc:407] Connection header received
0929 11:31:57.110045 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:57.110129 (+ 84us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:57.110263 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:57.110393 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:57.111093 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:57.112280 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:57.113594 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:57.114296 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:57.114437 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:57.115628 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:57.115640 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:57.115658 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:57.116489 (+ 831us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:57.116565 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:57.116591 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:57.116705 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:57.116863 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 11:31:57.116977 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 11:31:57.118155 11937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:57.156134 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:57.155652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39405 (local address 127.0.0.1:41718)
0929 11:31:57.155865 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:57.155969 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 11:31:57.157279 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:57.156907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39405 (local address 127.0.0.1:41732)
0929 11:31:57.157058 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:57.157138 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 11:31:57.166028 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:57.165429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39405 (local address 127.0.0.1:41748)
0929 11:31:57.165742 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:57.165838 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260929 11:31:57.167222 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:57.166790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39405 (local address 127.0.0.1:41752)
0929 11:31:57.166965 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:57.167078 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 11:31:57.206123 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:57.205582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39405 (local address 127.0.0.1:41754)
0929 11:31:57.205813 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:57.205937 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 11:31:57.207317 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:57.206888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39405 (local address 127.0.0.1:41766)
0929 11:31:57.207124 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:57.207198 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260929 11:31:57.233978 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 11815
I20260929 11:31:57.243899 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:57.243302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39405 (local address 127.0.0.1:41782)
0929 11:31:57.243549 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:57.243665 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 11:31:57.245293 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:57.244714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39405 (local address 127.0.0.1:41784)
0929 11:31:57.244953 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:57.245117 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 11:31:57.251051 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 11945
I20260929 11:31:57.258419 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:57.251906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51800 (local address 127.23.153.3:39249)
0929 11:31:57.252101 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:57.252106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:57.252126 (+ 20us) server_negotiation.cc:407] Connection header received
0929 11:31:57.252311 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:57.252315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:57.252415 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:57.252543 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:57.252985 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:57.253733 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:57.255297 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:57.255723 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:57.255836 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:57.257002 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:57.257011 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:57.257027 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:31:57.257716 (+ 689us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:57.257772 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:31:57.257800 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:57.257918 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:57.258060 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 11:31:57.258164 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 11:31:57.259240 12067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:57.308554 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:57.307998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39405 (local address 127.0.0.1:41794)
0929 11:31:57.308247 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:57.308348 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 11:31:57.309653 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:57.309273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39405 (local address 127.0.0.1:41810)
0929 11:31:57.309423 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:57.309515 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 11:31:57.347626 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:57.346875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39405 (local address 127.0.0.1:41818)
0929 11:31:57.347202 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:57.347313 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260929 11:31:57.348735 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:57.348387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39405 (local address 127.0.0.1:41826)
0929 11:31:57.348519 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:57.348602 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 11:31:57.354138 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 11945
I20260929 11:31:57.369565 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 11594
I20260929 11:31:57.371054 11655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:57.487360 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:57.486827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.1:39405 (local address 127.0.0.1:57914)
0929 11:31:57.487054 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 11:31:57.487157 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.1:39405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 11:31:57.487350 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 11594
2026-09-29T11:31:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3012 ms)
[----------] 6 tests from EncryptionPolicyTest (23229 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:31:57 dist-test-slave-qk05 krb5kdc[12126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:31:57 dist-test-slave-qk05 krb5kdc[12126](info): set up 2 sockets
Sep 29 11:31:57 dist-test-slave-qk05 krb5kdc[12126](info): commencing operation
krb5kdc: starting...
W20260929 11:31:58.551429 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:31:58 dist-test-slave-qk05 krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:31:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:31:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:31:58.826339 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:31:59.186434 12139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:31:59.186753 12139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:31:59.186846 12139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:31:59.194430 12139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:31:59.194567 12139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:31:59.194614 12139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:31:59.194674 12139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:31:59.194716 12139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:31:59.205058 12139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46427
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:31:59.206422 12139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:31:59.207830 12139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:31:59.219646 12144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:59.219646 12145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:31:59.219853 12147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:31:59.220669 12139 server_base.cc:1034] running on GCE node
I20260929 11:31:59.221531 12139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:31:59.223124 12139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:31:59.224356 12139 hybrid_clock.cc:648] HybridClock initialized: now 1790681519224303 us; error 51 us; skew 500 ppm
I20260929 11:31:59.224740 12139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:31:59 dist-test-slave-qk05 krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:31:59.228257 12139 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:31:59.230203 12139 webserver.cc:460] Webserver started at http://127.0.0.1:34301/ using document root <none> and password file <none>
I20260929 11:31:59.231092 12139 fs_manager.cc:359] Metadata directory not provided
I20260929 11:31:59.231245 12139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:31:59.231580 12139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:31:59.234978 12139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "e85f05eb89fe44f28e47497b2b0bf64b"
format_stamp: "Formatted at 2026-09-29 11:31:59 on dist-test-slave-qk05"
server_key: "0e5c7989170680aacd02a55ea93fa585"
server_key_iv: "fcba289593c12432cbbf4d8ad777724c"
server_key_version: "encryptionkey@0"
I20260929 11:31:59.235904 12139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "e85f05eb89fe44f28e47497b2b0bf64b"
format_stamp: "Formatted at 2026-09-29 11:31:59 on dist-test-slave-qk05"
server_key: "0e5c7989170680aacd02a55ea93fa585"
server_key_iv: "fcba289593c12432cbbf4d8ad777724c"
server_key_version: "encryptionkey@0"
I20260929 11:31:59.241888 12139 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:31:59.246220 12154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:31:59.247996 12139 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:31:59.248194 12139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "e85f05eb89fe44f28e47497b2b0bf64b"
format_stamp: "Formatted at 2026-09-29 11:31:59 on dist-test-slave-qk05"
server_key: "0e5c7989170680aacd02a55ea93fa585"
server_key_iv: "fcba289593c12432cbbf4d8ad777724c"
server_key_version: "encryptionkey@0"
I20260929 11:31:59.248423 12139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:31:59.291158 12139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:31:59.296247 12139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:31:59.296604 12139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:31:59.317276 12139 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46427
I20260929 11:31:59.317284 12205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46427 every 8 connection(s)
I20260929 11:31:59.319936 12139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:31:59.324599 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 12139
I20260929 11:31:59.324823 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:31:59.325630 12206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:31:59.325665 24164 external_mini_cluster.cc:1428] Setting key 247653a33d2caa80e7288f7483158faf
I20260929 11:31:59.342041 12206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b: Bootstrap starting.
Sep 29 11:31:59 dist-test-slave-qk05 krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 11:31:59.345681 12206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b: Neither blocks nor log segments found. Creating new log.
I20260929 11:31:59.347319 12206 log.cc:824] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b: Log is configured to *not* fsync() on all Append() calls
I20260929 11:31:59.351877 12206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b: No bootstrap required, opened a new log
I20260929 11:31:59.354149 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:59.327913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54927 (local address 127.0.0.1:46427)
0929 11:31:59.328783 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:59.328797 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:59.328849 (+ 52us) server_negotiation.cc:407] Connection header received
0929 11:31:59.330848 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:59.330876 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:59.331477 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:59.331929 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:59.334068 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:59.335650 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:59.337699 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:59.338602 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:59.338797 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:59.344288 (+ 5491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:59.344301 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:59.344325 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:59.344394 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:59.348529 (+ 4135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:59.349750 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:59.349759 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:59.349767 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:59.349984 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:59.350736 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:59.350740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:59.350742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:59.351377 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:59.351653 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:59.352368 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 11:31:59.352937 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":388,"threads_started":1}
I20260929 11:31:59.359386 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:31:59.361518 12206 raft_consensus.cc:348] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e85f05eb89fe44f28e47497b2b0bf64b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46427 } }
I20260929 11:31:59.361965 12206 raft_consensus.cc:374] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:31:59.362067 12206 raft_consensus.cc:729] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85f05eb89fe44f28e47497b2b0bf64b, State: Initialized, Role: FOLLOWER
I20260929 11:31:59.362742 12206 consensus_queue.cc:260] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e85f05eb89fe44f28e47497b2b0bf64b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46427 } }
I20260929 11:31:59.363035 12206 raft_consensus.cc:388] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:31:59.363199 12206 raft_consensus.cc:482] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:31:59.363363 12206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:31:59.366120 12206 raft_consensus.cc:504] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e85f05eb89fe44f28e47497b2b0bf64b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46427 } }
I20260929 11:31:59.366704 12206 leader_election.cc:302] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e85f05eb89fe44f28e47497b2b0bf64b; no voters:
I20260929 11:31:59.367381 12206 leader_election.cc:288] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:31:59.367529 12216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:31:59.368755 12216 raft_consensus.cc:686] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b [term 1 LEADER]: Becoming Leader. State: Replica: e85f05eb89fe44f28e47497b2b0bf64b, State: Running, Role: LEADER
I20260929 11:31:59.369351 12216 consensus_queue.cc:237] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e85f05eb89fe44f28e47497b2b0bf64b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46427 } }
I20260929 11:31:59.370009 12206 sys_catalog.cc:564] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:31:59.373220 12217 sys_catalog.cc:455] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e85f05eb89fe44f28e47497b2b0bf64b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e85f05eb89fe44f28e47497b2b0bf64b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46427 } } }
I20260929 11:31:59.374495 12217 sys_catalog.cc:458] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b [sys.catalog]: This master's current role is: LEADER
I20260929 11:31:59.374505 12218 sys_catalog.cc:455] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e85f05eb89fe44f28e47497b2b0bf64b. Latest consensus state: current_term: 1 leader_uuid: "e85f05eb89fe44f28e47497b2b0bf64b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e85f05eb89fe44f28e47497b2b0bf64b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46427 } } }
I20260929 11:31:59.375368 12218 sys_catalog.cc:458] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b [sys.catalog]: This master's current role is: LEADER
I20260929 11:31:59.384219 12223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:31:59.390002 12223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:31:59.390194 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:31:59.364649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34603 (local address 127.0.0.1:46427)
0929 11:31:59.364860 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 11:31:59.364867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:31:59.365292 (+ 425us) server_negotiation.cc:407] Connection header received
0929 11:31:59.365430 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:31:59.365436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:31:59.365526 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:31:59.365646 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:31:59.367289 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:59.368666 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:59.370664 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:31:59.371434 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:31:59.371601 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:31:59.375245 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:31:59.375256 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:31:59.375262 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:31:59.375315 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:31:59.385763 (+ 10448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:59.387091 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:59.387097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:59.387101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:59.387279 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:31:59.387949 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:31:59.387954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:31:59.387958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:31:59.388458 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:31:59.388740 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 11:31:59.389503 (+ 763us) server_negotiation.cc:299] Negotiation successful
0929 11:31:59.389747 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:31:59.404839 12223 catalog_manager.cc:1347] Generated new cluster ID: 7166c47e31fd4bdbb60a5a849d390a32
I20260929 11:31:59.404981 12223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:31:59.429284 12223 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:31:59.431406 12223 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:31:59.441385 12223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e85f05eb89fe44f28e47497b2b0bf64b: Generated new TSK 0
I20260929 11:31:59.442602 12223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:31:59.458273 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 12139
I20260929 11:31:59.460441 12201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:31:59.561769 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 12139
W20260929 11:31:59.577410 12124 connection.cc:535] client connection to 127.0.0.1:46427 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46427): Connection reset by peer (error 104)
2026-09-29T11:31:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2066 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:31:59 dist-test-slave-qk05 krb5kdc[12238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:31:59 dist-test-slave-qk05 krb5kdc[12238](info): set up 2 sockets
Sep 29 11:31:59 dist-test-slave-qk05 krb5kdc[12238](info): commencing operation
krb5kdc: starting...
W20260929 11:32:00.603503 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:00 dist-test-slave-qk05 krb5kdc[12238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:32:00.864512 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44895
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:01.226482 12251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:01.226832 12251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:01.226970 12251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:01.235148 12251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:01.235314 12251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:01.235391 12251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:01.235548 12251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:01.235683 12251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:01.246565 12251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44895
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:01.248188 12251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:01.249936 12251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:01.262866 12256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:01.262780 12257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:01.262828 12259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:01.263916 12251 server_base.cc:1034] running on GCE node
I20260929 11:32:01.264865 12251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:01.266630 12251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:01.267905 12251 hybrid_clock.cc:648] HybridClock initialized: now 1790681521267848 us; error 47 us; skew 500 ppm
I20260929 11:32:01.268316 12251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:01 dist-test-slave-qk05 krb5kdc[12238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:01.277035 12251 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:32:01.279922 12251 webserver.cc:460] Webserver started at http://127.0.0.1:37491/ using document root <none> and password file <none>
I20260929 11:32:01.281083 12251 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:01.281258 12251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:01.281697 12251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:01.286958 12251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "a6b0768d72b346ef83208fcba3f494cc"
format_stamp: "Formatted at 2026-09-29 11:32:01 on dist-test-slave-qk05"
server_key: "6b4ceab8e98b8aded0b8951d5f2b4b7b"
server_key_iv: "c2d0a3fb5e8f2d6c2944573b3f8385b7"
server_key_version: "encryptionkey@0"
I20260929 11:32:01.288054 12251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "a6b0768d72b346ef83208fcba3f494cc"
format_stamp: "Formatted at 2026-09-29 11:32:01 on dist-test-slave-qk05"
server_key: "6b4ceab8e98b8aded0b8951d5f2b4b7b"
server_key_iv: "c2d0a3fb5e8f2d6c2944573b3f8385b7"
server_key_version: "encryptionkey@0"
I20260929 11:32:01.294677 12251 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 11:32:01.299063 12266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:01.300989 12251 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:32:01.301222 12251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "a6b0768d72b346ef83208fcba3f494cc"
format_stamp: "Formatted at 2026-09-29 11:32:01 on dist-test-slave-qk05"
server_key: "6b4ceab8e98b8aded0b8951d5f2b4b7b"
server_key_iv: "c2d0a3fb5e8f2d6c2944573b3f8385b7"
server_key_version: "encryptionkey@0"
I20260929 11:32:01.301530 12251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:01.317530 12251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:01.322044 12251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:01.322321 12251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:01.340793 12251 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44895
I20260929 11:32:01.340834 12317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44895 every 8 connection(s)
I20260929 11:32:01.342291 12251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:01.344590 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 12251
I20260929 11:32:01.344775 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:01.345328 24164 external_mini_cluster.cc:1428] Setting key 4166c092c3a1a0f4fa92bf3775016151
I20260929 11:32:01.347114 12318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:01.360188 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:01.347230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47389 (local address 127.0.0.1:44895)
0929 11:32:01.348078 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:01.348090 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:01.348135 (+ 45us) server_negotiation.cc:407] Connection header received
0929 11:32:01.349849 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:01.349873 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:01.350356 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:01.350910 (+ 554us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:01.351646 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:01.353207 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:01.355177 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:01.355952 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:01.356113 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:01.357537 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:01.357546 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:01.357564 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:01.358231 (+ 667us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:01.358327 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:01.358378 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:01.358531 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:01.358837 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 11:32:01.359103 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":389,"threads_started":1}
I20260929 11:32:01.361110 12318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc: Bootstrap starting.
I20260929 11:32:01.364686 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:01.364717 12318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:01.365787 12318 log.cc:824] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:01.368794 12318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc: No bootstrap required, opened a new log
I20260929 11:32:01.376747 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:01.369114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49751 (local address 127.0.0.1:44895)
0929 11:32:01.369378 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:01.369386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:01.369904 (+ 518us) server_negotiation.cc:407] Connection header received
0929 11:32:01.370058 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:01.370064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:01.370146 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:01.370270 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:01.370764 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:01.372106 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:01.373599 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:01.374315 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:01.374475 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:01.375894 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:01.375902 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:01.375907 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:01.375935 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:01.376007 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:01.376037 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:01.376146 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:01.376323 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 11:32:01.376396 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:32:01.378962 12318 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6b0768d72b346ef83208fcba3f494cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44895 } }
I20260929 11:32:01.379432 12318 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:01.379606 12318 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6b0768d72b346ef83208fcba3f494cc, State: Initialized, Role: FOLLOWER
I20260929 11:32:01.380192 12318 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6b0768d72b346ef83208fcba3f494cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44895 } }
I20260929 11:32:01.380421 12318 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:01.380533 12318 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:01.380657 12318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:01.382879 12318 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6b0768d72b346ef83208fcba3f494cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44895 } }
I20260929 11:32:01.383538 12318 leader_election.cc:302] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6b0768d72b346ef83208fcba3f494cc; no voters:
I20260929 11:32:01.384228 12318 leader_election.cc:288] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:01.384364 12328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:01.385036 12328 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc [term 1 LEADER]: Becoming Leader. State: Replica: a6b0768d72b346ef83208fcba3f494cc, State: Running, Role: LEADER
I20260929 11:32:01.385639 12328 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6b0768d72b346ef83208fcba3f494cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44895 } }
I20260929 11:32:01.386314 12318 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:01.388850 12329 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6b0768d72b346ef83208fcba3f494cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6b0768d72b346ef83208fcba3f494cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44895 } } }
I20260929 11:32:01.388881 12330 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6b0768d72b346ef83208fcba3f494cc. Latest consensus state: current_term: 1 leader_uuid: "a6b0768d72b346ef83208fcba3f494cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6b0768d72b346ef83208fcba3f494cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44895 } } }
I20260929 11:32:01.389501 12329 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:01.389501 12330 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:01.390580 12337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:01.406064 12337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:01.417739 12337 catalog_manager.cc:1347] Generated new cluster ID: 8b691738b1f445778b6251c724a1136f
I20260929 11:32:01.417928 12337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:01.444269 12337 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:01.445991 12337 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:01.458180 12337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6b0768d72b346ef83208fcba3f494cc: Generated new TSK 0
I20260929 11:32:01.459378 12337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:32:01.470924 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 12251
W20260929 11:32:01.473245 12313 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47389
W20260929 11:32:01.474143 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 12251: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:32:01.474354 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 12251
W20260929 11:32:01.489163 12236 connection.cc:535] client connection to 127.0.0.1:44895 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44895): Connection reset by peer (error 104)
2026-09-29T11:32:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1911 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:01 dist-test-slave-qk05 krb5kdc[12349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:01 dist-test-slave-qk05 krb5kdc[12349](info): set up 2 sockets
Sep 29 11:32:01 dist-test-slave-qk05 krb5kdc[12349](info): commencing operation
krb5kdc: starting...
W20260929 11:32:02.539629 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:02 dist-test-slave-qk05 krb5kdc[12349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:32:02.798774 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:03.181588 12362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:03.181973 12362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:03.182116 12362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:03.190438 12362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:03.190619 12362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:03.190696 12362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:03.190786 12362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:03.190848 12362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:03.202832 12362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40769
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:03.204352 12362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:03.205792 12362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:03.219609 12368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:03.219725 12367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:03.220175 12370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:03.220595 12362 server_base.cc:1034] running on GCE node
I20260929 11:32:03.221642 12362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:03.223524 12362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:03.224961 12362 hybrid_clock.cc:648] HybridClock initialized: now 1790681523224879 us; error 65 us; skew 500 ppm
I20260929 11:32:03.225380 12362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:03 dist-test-slave-qk05 krb5kdc[12349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:03.229084 12362 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:32:03.231397 12362 webserver.cc:460] Webserver started at http://127.0.0.1:34411/ using document root <none> and password file <none>
I20260929 11:32:03.232177 12362 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:03.232306 12362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:03.232594 12362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:03.235927 12362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "7a5bfcd69a514d64965184e80623c149"
format_stamp: "Formatted at 2026-09-29 11:32:03 on dist-test-slave-qk05"
server_key: "327122710bf867802aa927417e7514e5"
server_key_iv: "2096981ddf49a06adf3391c7e3ee4560"
server_key_version: "encryptionkey@0"
I20260929 11:32:03.236789 12362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "7a5bfcd69a514d64965184e80623c149"
format_stamp: "Formatted at 2026-09-29 11:32:03 on dist-test-slave-qk05"
server_key: "327122710bf867802aa927417e7514e5"
server_key_iv: "2096981ddf49a06adf3391c7e3ee4560"
server_key_version: "encryptionkey@0"
I20260929 11:32:03.244591 12362 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260929 11:32:03.249691 12377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:03.251632 12362 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 11:32:03.251952 12362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "7a5bfcd69a514d64965184e80623c149"
format_stamp: "Formatted at 2026-09-29 11:32:03 on dist-test-slave-qk05"
server_key: "327122710bf867802aa927417e7514e5"
server_key_iv: "2096981ddf49a06adf3391c7e3ee4560"
server_key_version: "encryptionkey@0"
I20260929 11:32:03.252280 12362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:03.261319 12362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:03.267441 12362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:03.268322 12362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:03.290654 12362 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40769
I20260929 11:32:03.290691 12428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40769 every 8 connection(s)
I20260929 11:32:03.292402 12362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:03.296604 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 12362
I20260929 11:32:03.296860 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:03.297551 24164 external_mini_cluster.cc:1428] Setting key 185b085b21d24daa00830d6b545f3ecf
I20260929 11:32:03.299130 12429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:03.306861 12432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:03.300042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51631 (local address 127.0.0.1:40769)
0929 11:32:03.300977 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:03.300990 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:03.301039 (+ 49us) server_negotiation.cc:407] Connection header received
0929 11:32:03.302771 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:03.302802 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:03.303310 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:03.303763 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:03.304939 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:03.304947 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:03.304963 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:03.304979 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:03.305064 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:03.305085 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:03.305334 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:03.305513 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 11:32:03.305745 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":410,"threads_started":1}
I20260929 11:32:03.312718 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:03.313421 12429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149: Bootstrap starting.
I20260929 11:32:03.317246 12429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:03.318958 12429 log.cc:824] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:03.321774 12432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:03.317947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56201 (local address 127.0.0.1:40769)
0929 11:32:03.318414 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:03.318437 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:03.318817 (+ 380us) server_negotiation.cc:407] Connection header received
0929 11:32:03.319054 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:03.319062 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:03.319325 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:03.319485 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:03.320999 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:03.321006 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:03.321012 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:03.321041 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:03.321107 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:03.321128 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:03.321260 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:03.321393 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 11:32:03.321475 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260929 11:32:03.322384 12429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149: No bootstrap required, opened a new log
I20260929 11:32:03.330868 12429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5bfcd69a514d64965184e80623c149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40769 } }
I20260929 11:32:03.331403 12429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:03.331488 12429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5bfcd69a514d64965184e80623c149, State: Initialized, Role: FOLLOWER
I20260929 11:32:03.332124 12429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5bfcd69a514d64965184e80623c149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40769 } }
I20260929 11:32:03.332382 12429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:03.332517 12429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:03.332679 12429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:03.335044 12429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5bfcd69a514d64965184e80623c149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40769 } }
I20260929 11:32:03.335729 12429 leader_election.cc:302] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a5bfcd69a514d64965184e80623c149; no voters:
I20260929 11:32:03.336429 12429 leader_election.cc:288] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:03.336551 12439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:03.337175 12439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149 [term 1 LEADER]: Becoming Leader. State: Replica: 7a5bfcd69a514d64965184e80623c149, State: Running, Role: LEADER
I20260929 11:32:03.337833 12439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5bfcd69a514d64965184e80623c149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40769 } }
I20260929 11:32:03.339231 12429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:03.341264 12440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a5bfcd69a514d64965184e80623c149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5bfcd69a514d64965184e80623c149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40769 } } }
I20260929 11:32:03.341538 12441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a5bfcd69a514d64965184e80623c149. Latest consensus state: current_term: 1 leader_uuid: "7a5bfcd69a514d64965184e80623c149" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5bfcd69a514d64965184e80623c149" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40769 } } }
I20260929 11:32:03.341799 12440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:03.341856 12441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:03.344553 12445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:03.353106 12445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:03.364756 12445 catalog_manager.cc:1347] Generated new cluster ID: dfcfddef38444691a9f40a897ecca3b9
I20260929 11:32:03.364908 12445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:03.383560 12445 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:03.384084 12445 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:03.392743 12445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a5bfcd69a514d64965184e80623c149: Generated new TSK 0
I20260929 11:32:03.393726 12445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:32:03.398391 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 12362
W20260929 11:32:03.399721 12424 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51631
W20260929 11:32:03.400605 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 12362: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:32:03.400769 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 12362
2026-09-29T11:32:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1929 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:03 dist-test-slave-qk05 krb5kdc[12460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:03 dist-test-slave-qk05 krb5kdc[12460](info): set up 2 sockets
Sep 29 11:32:03 dist-test-slave-qk05 krb5kdc[12460](info): commencing operation
krb5kdc: starting...
W20260929 11:32:04.467283 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.002s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:04 dist-test-slave-qk05 krb5kdc[12460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:32:04.738188 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40577
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:05.112955 12473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:05.113386 12473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:05.113502 12473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:05.122390 12473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:05.122560 12473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:05.122612 12473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:05.122676 12473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:05.122726 12473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:05.134639 12473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:05.136093 12473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:05.137476 12473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:05.150771 12478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:05.150738 12479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:05.150806 12481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:05.152297 12473 server_base.cc:1034] running on GCE node
I20260929 11:32:05.153183 12473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:05.154925 12473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:05.156319 12473 hybrid_clock.cc:648] HybridClock initialized: now 1790681525156249 us; error 55 us; skew 500 ppm
I20260929 11:32:05.156879 12473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:05 dist-test-slave-qk05 krb5kdc[12460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:05.160444 12473 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:32:05.162209 12473 webserver.cc:460] Webserver started at http://127.0.0.1:36475/ using document root <none> and password file <none>
I20260929 11:32:05.162984 12473 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:05.163172 12473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:05.163496 12473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:05.167133 12473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "538cf8b54dc0461fb45263e5f146de1e"
format_stamp: "Formatted at 2026-09-29 11:32:05 on dist-test-slave-qk05"
server_key: "c24b23d2bf48bec70eeefd80b8b1ffbb"
server_key_iv: "d0095443667a0e08780560b23c9ffac1"
server_key_version: "encryptionkey@0"
I20260929 11:32:05.168128 12473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "538cf8b54dc0461fb45263e5f146de1e"
format_stamp: "Formatted at 2026-09-29 11:32:05 on dist-test-slave-qk05"
server_key: "c24b23d2bf48bec70eeefd80b8b1ffbb"
server_key_iv: "d0095443667a0e08780560b23c9ffac1"
server_key_version: "encryptionkey@0"
I20260929 11:32:05.174305 12473 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 11:32:05.178516 12488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:05.180168 12473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:32:05.180392 12473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "538cf8b54dc0461fb45263e5f146de1e"
format_stamp: "Formatted at 2026-09-29 11:32:05 on dist-test-slave-qk05"
server_key: "c24b23d2bf48bec70eeefd80b8b1ffbb"
server_key_iv: "d0095443667a0e08780560b23c9ffac1"
server_key_version: "encryptionkey@0"
I20260929 11:32:05.180614 12473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:05.197680 12473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:05.202500 12473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:05.202826 12473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:05.223601 12473 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40577
I20260929 11:32:05.223740 12539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40577 every 8 connection(s)
I20260929 11:32:05.225153 12473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:05.226307 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 12473
I20260929 11:32:05.226495 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:05.226969 24164 external_mini_cluster.cc:1428] Setting key e86109f8956294ed24c4d7aa929bd591
I20260929 11:32:05.229889 12540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:05.242911 12540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e: Bootstrap starting.
Sep 29 11:32:05 dist-test-slave-qk05 krb5kdc[12460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 11:32:05.247285 12540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:05.249142 12540 log.cc:824] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:05.253518 12540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e: No bootstrap required, opened a new log
I20260929 11:32:05.255164 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:05.228928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60893 (local address 127.0.0.1:40577)
0929 11:32:05.230165 (+ 1237us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:05.230194 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:05.230264 (+ 70us) server_negotiation.cc:407] Connection header received
0929 11:32:05.232905 (+ 2641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:05.232939 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:05.233646 (+ 707us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:05.234473 (+ 827us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:05.236178 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:05.237658 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:05.239421 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:05.240252 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:05.240477 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:05.244980 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:05.244998 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:05.245021 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:05.245088 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:05.249670 (+ 4582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:05.250970 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:05.250979 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:05.250990 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:05.251259 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:05.251689 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:05.251695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:05.251697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:05.252442 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:05.252751 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:05.253202 (+ 451us) server_negotiation.cc:299] Negotiation successful
0929 11:32:05.253706 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":469,"threads_started":1}
I20260929 11:32:05.260414 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:05.263435 12540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538cf8b54dc0461fb45263e5f146de1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40577 } }
I20260929 11:32:05.263994 12540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:05.264119 12540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538cf8b54dc0461fb45263e5f146de1e, State: Initialized, Role: FOLLOWER
I20260929 11:32:05.264914 12540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538cf8b54dc0461fb45263e5f146de1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40577 } }
I20260929 11:32:05.265226 12540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:05.265396 12540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:05.265580 12540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:05.268976 12540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538cf8b54dc0461fb45263e5f146de1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40577 } }
I20260929 11:32:05.269588 12540 leader_election.cc:302] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 538cf8b54dc0461fb45263e5f146de1e; no voters:
I20260929 11:32:05.270280 12540 leader_election.cc:288] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:05.270390 12550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:05.271638 12550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e [term 1 LEADER]: Becoming Leader. State: Replica: 538cf8b54dc0461fb45263e5f146de1e, State: Running, Role: LEADER
I20260929 11:32:05.272374 12550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538cf8b54dc0461fb45263e5f146de1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40577 } }
I20260929 11:32:05.272957 12540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:05.275641 12552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 538cf8b54dc0461fb45263e5f146de1e. Latest consensus state: current_term: 1 leader_uuid: "538cf8b54dc0461fb45263e5f146de1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538cf8b54dc0461fb45263e5f146de1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40577 } } }
I20260929 11:32:05.276141 12552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:05.276168 12551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "538cf8b54dc0461fb45263e5f146de1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "538cf8b54dc0461fb45263e5f146de1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40577 } } }
I20260929 11:32:05.277947 12551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:05.284512 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:05.265970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52869 (local address 127.0.0.1:40577)
0929 11:32:05.266200 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:05.266206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:05.266562 (+ 356us) server_negotiation.cc:407] Connection header received
0929 11:32:05.266741 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:05.266745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:05.266821 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:05.266919 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:05.268825 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:05.269647 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:05.271401 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:05.272078 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:05.272205 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:05.275174 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:05.275181 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:05.275186 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:05.275233 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:05.279270 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:05.280467 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:05.280474 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:05.280479 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:05.280690 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:05.281269 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:05.281275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:05.281278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:05.281768 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:05.282078 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:05.283691 (+ 1613us) server_negotiation.cc:299] Negotiation successful
0929 11:32:05.284021 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 11:32:05.290066 12555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:05.295502 12555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:05.306119 12555 catalog_manager.cc:1347] Generated new cluster ID: 1c00b4ab37bf4c76abafc7f50cb0a9fc
I20260929 11:32:05.306262 12555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:05.325977 12555 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:05.328642 12555 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:05.336603 12555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 538cf8b54dc0461fb45263e5f146de1e: Generated new TSK 0
I20260929 11:32:05.337595 12555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:32:05.351336 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 12473
I20260929 11:32:05.352869 12535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:05.453935 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 12473
2026-09-29T11:32:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2056 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:05 dist-test-slave-qk05 krb5kdc[12572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:05 dist-test-slave-qk05 krb5kdc[12572](info): set up 2 sockets
Sep 29 11:32:05 dist-test-slave-qk05 krb5kdc[12572](info): commencing operation
krb5kdc: starting...
W20260929 11:32:06.511400 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:06 dist-test-slave-qk05 krb5kdc[12572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:32:06.799578 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:07.177742 12585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:07.178062 12585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:07.178158 12585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:07.185667 12585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:07.185808 12585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:07.185858 12585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:07.185911 12585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:07.185994 12585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:07.197165 12585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:07.198553 12585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:07.199961 12585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:07.212668 12591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:07.212739 12590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:07.212868 12593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:07.213339 12585 server_base.cc:1034] running on GCE node
I20260929 11:32:07.214257 12585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:07.215921 12585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:07.217170 12585 hybrid_clock.cc:648] HybridClock initialized: now 1790681527217122 us; error 48 us; skew 500 ppm
I20260929 11:32:07.217562 12585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:07 dist-test-slave-qk05 krb5kdc[12572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:07.225657 12585 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:32:07.227497 12585 webserver.cc:460] Webserver started at http://127.0.0.1:33845/ using document root <none> and password file <none>
I20260929 11:32:07.228261 12585 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:07.228386 12585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:07.228699 12585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:07.232025 12585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "efa1f05c74374ef788745495f5916535"
format_stamp: "Formatted at 2026-09-29 11:32:07 on dist-test-slave-qk05"
server_key: "9bfa3366ccdebbfc2563803e6fb82985"
server_key_iv: "02d530e311c96489237b0e53313f5595"
server_key_version: "encryptionkey@0"
I20260929 11:32:07.232933 12585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "efa1f05c74374ef788745495f5916535"
format_stamp: "Formatted at 2026-09-29 11:32:07 on dist-test-slave-qk05"
server_key: "9bfa3366ccdebbfc2563803e6fb82985"
server_key_iv: "02d530e311c96489237b0e53313f5595"
server_key_version: "encryptionkey@0"
I20260929 11:32:07.239162 12585 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 11:32:07.243299 12600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:07.244966 12585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 11:32:07.245174 12585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "efa1f05c74374ef788745495f5916535"
format_stamp: "Formatted at 2026-09-29 11:32:07 on dist-test-slave-qk05"
server_key: "9bfa3366ccdebbfc2563803e6fb82985"
server_key_iv: "02d530e311c96489237b0e53313f5595"
server_key_version: "encryptionkey@0"
I20260929 11:32:07.245397 12585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:07.263499 12585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:07.268725 12585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:07.269040 12585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:07.288753 12585 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45927
I20260929 11:32:07.288810 12651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45927 every 8 connection(s)
I20260929 11:32:07.290427 12585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:07.294721 12652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:07.297013 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 12585
I20260929 11:32:07.297219 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:07.297736 24164 external_mini_cluster.cc:1428] Setting key b1d0194ce6f491d60f49aa14459203af
I20260929 11:32:07.312187 12652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535: Bootstrap starting.
I20260929 11:32:07.313428 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:07.299644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46067 (local address 127.0.0.1:45927)
0929 11:32:07.300517 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:07.300529 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:07.300574 (+ 45us) server_negotiation.cc:407] Connection header received
0929 11:32:07.302302 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:07.302331 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:07.302844 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:07.303291 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:07.303942 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:07.305288 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:07.307683 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:07.308760 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:07.308952 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:07.310769 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:07.310778 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:07.310797 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:07.311463 (+ 666us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:07.311557 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:07.311593 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:07.311712 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:07.312000 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 11:32:07.312304 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":386,"threads_started":1}
I20260929 11:32:07.315953 12652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:07.317394 12652 log.cc:824] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:07.317770 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:07.320604 12652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535: No bootstrap required, opened a new log
I20260929 11:32:07.329974 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:07.322412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56695 (local address 127.0.0.1:45927)
0929 11:32:07.322659 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:07.322668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:07.323124 (+ 456us) server_negotiation.cc:407] Connection header received
0929 11:32:07.323318 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:07.323322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:07.323497 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:07.323605 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:07.324253 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:07.325055 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:07.327293 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:07.327814 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:07.327963 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:07.329223 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:07.329228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:07.329231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:07.329254 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:07.329303 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:07.329324 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:07.329404 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:07.329552 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 11:32:07.329644 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 11:32:07.331203 12652 raft_consensus.cc:348] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa1f05c74374ef788745495f5916535" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260929 11:32:07.331702 12652 raft_consensus.cc:374] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:07.331809 12652 raft_consensus.cc:729] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa1f05c74374ef788745495f5916535, State: Initialized, Role: FOLLOWER
I20260929 11:32:07.332463 12652 consensus_queue.cc:260] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa1f05c74374ef788745495f5916535" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260929 11:32:07.332721 12652 raft_consensus.cc:388] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:07.332863 12652 raft_consensus.cc:482] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:07.333034 12652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:07.335449 12652 raft_consensus.cc:504] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa1f05c74374ef788745495f5916535" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260929 11:32:07.336031 12652 leader_election.cc:302] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efa1f05c74374ef788745495f5916535; no voters:
I20260929 11:32:07.336735 12652 leader_election.cc:288] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:07.336886 12662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:07.338279 12662 raft_consensus.cc:686] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535 [term 1 LEADER]: Becoming Leader. State: Replica: efa1f05c74374ef788745495f5916535, State: Running, Role: LEADER
I20260929 11:32:07.338929 12652 sys_catalog.cc:564] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:07.339082 12662 consensus_queue.cc:237] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa1f05c74374ef788745495f5916535" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260929 11:32:07.342845 12664 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efa1f05c74374ef788745495f5916535. Latest consensus state: current_term: 1 leader_uuid: "efa1f05c74374ef788745495f5916535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa1f05c74374ef788745495f5916535" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } } }
I20260929 11:32:07.343392 12664 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:07.344117 12663 sys_catalog.cc:455] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efa1f05c74374ef788745495f5916535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efa1f05c74374ef788745495f5916535" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } } }
I20260929 11:32:07.344455 12663 sys_catalog.cc:458] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:07.348621 12672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:07.354595 12672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:07.366379 12672 catalog_manager.cc:1347] Generated new cluster ID: 4bb9a62ab8e94f4f8ef12e1088fed119
I20260929 11:32:07.366521 12672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:07.377770 12672 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:07.379614 12672 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:07.386775 12672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efa1f05c74374ef788745495f5916535: Generated new TSK 0
I20260929 11:32:07.388018 12672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:32:07.403076 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 12585
W20260929 11:32:07.404317 12647 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46067
W20260929 11:32:07.405187 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 12585: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:32:07.405329 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 12585
2026-09-29T11:32:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1945 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:07 dist-test-slave-qk05 krb5kdc[12683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:07 dist-test-slave-qk05 krb5kdc[12683](info): set up 2 sockets
Sep 29 11:32:07 dist-test-slave-qk05 krb5kdc[12683](info): commencing operation
krb5kdc: starting...
W20260929 11:32:08.447393 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:08 dist-test-slave-qk05 krb5kdc[12683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:32:08.713976 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41557
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:09.071517 12696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:09.071951 12696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:09.072090 12696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:09.081454 12696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:09.081624 12696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:09.081681 12696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:09.081772 12696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:09.081835 12696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:09.094175 12696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:09.095710 12696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:09.097186 12696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:09.112280 12702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:09.112353 12704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:09.112323 12701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:09.113230 12696 server_base.cc:1034] running on GCE node
I20260929 11:32:09.114102 12696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:09.115739 12696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:09.117189 12696 hybrid_clock.cc:648] HybridClock initialized: now 1790681529117072 us; error 107 us; skew 500 ppm
I20260929 11:32:09.117766 12696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:09 dist-test-slave-qk05 krb5kdc[12683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:09.121712 12696 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:32:09.124119 12696 webserver.cc:460] Webserver started at http://127.0.0.1:42279/ using document root <none> and password file <none>
I20260929 11:32:09.125255 12696 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:09.125974 12696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:09.126470 12696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:09.130137 12696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "9b1ac4e4f7444e9bb3da87b90fdca90c"
format_stamp: "Formatted at 2026-09-29 11:32:09 on dist-test-slave-qk05"
server_key: "97025460870188d7733180bac99857ca"
server_key_iv: "bac312f33b2e08e5c374df923649eaa8"
server_key_version: "encryptionkey@0"
I20260929 11:32:09.131047 12696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "9b1ac4e4f7444e9bb3da87b90fdca90c"
format_stamp: "Formatted at 2026-09-29 11:32:09 on dist-test-slave-qk05"
server_key: "97025460870188d7733180bac99857ca"
server_key_iv: "bac312f33b2e08e5c374df923649eaa8"
server_key_version: "encryptionkey@0"
I20260929 11:32:09.136976 12696 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 11:32:09.141140 12711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:09.142707 12696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:32:09.142915 12696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "9b1ac4e4f7444e9bb3da87b90fdca90c"
format_stamp: "Formatted at 2026-09-29 11:32:09 on dist-test-slave-qk05"
server_key: "97025460870188d7733180bac99857ca"
server_key_iv: "bac312f33b2e08e5c374df923649eaa8"
server_key_version: "encryptionkey@0"
I20260929 11:32:09.143170 12696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:09.149828 12696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:09.154578 12696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:09.154901 12696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:09.175280 12696 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41557
I20260929 11:32:09.175314 12762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41557 every 8 connection(s)
I20260929 11:32:09.176784 12696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:09.181180 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 12696
I20260929 11:32:09.181342 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:09.181576 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:09.181830 24164 external_mini_cluster.cc:1428] Setting key bd287e4aad2ba2fd591baa90e3b27de0
I20260929 11:32:09.191402 12766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:09.184147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35909 (local address 127.0.0.1:41557)
0929 11:32:09.185144 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:09.185166 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:09.185228 (+ 62us) server_negotiation.cc:407] Connection header received
0929 11:32:09.187087 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:09.187129 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:09.187679 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:09.188231 (+ 552us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:09.189291 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:09.189300 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:09.189319 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:09.189338 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:09.189415 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:09.189442 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:09.189686 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:09.189837 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 11:32:09.190062 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":398,"threads_started":1}
I20260929 11:32:09.195823 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:09.197228 12763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c: Bootstrap starting.
I20260929 11:32:09.201969 12763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:09.203439 12763 log.cc:824] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:09.204314 12766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:09.201595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47729 (local address 127.0.0.1:41557)
0929 11:32:09.201792 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:09.201798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:09.202081 (+ 283us) server_negotiation.cc:407] Connection header received
0929 11:32:09.202263 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:09.202268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:09.202349 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:09.202507 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:09.203512 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:09.203518 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:09.203522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:09.203542 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:09.203595 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:09.203617 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:09.203744 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:09.203901 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 11:32:09.203981 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 11:32:09.207072 12763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c: No bootstrap required, opened a new log
I20260929 11:32:09.215806 12763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b1ac4e4f7444e9bb3da87b90fdca90c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41557 } }
I20260929 11:32:09.216527 12763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:09.216681 12763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b1ac4e4f7444e9bb3da87b90fdca90c, State: Initialized, Role: FOLLOWER
I20260929 11:32:09.217379 12763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b1ac4e4f7444e9bb3da87b90fdca90c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41557 } }
I20260929 11:32:09.217634 12763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:09.217774 12763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:09.217928 12763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:09.220366 12763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b1ac4e4f7444e9bb3da87b90fdca90c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41557 } }
I20260929 11:32:09.220989 12763 leader_election.cc:302] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b1ac4e4f7444e9bb3da87b90fdca90c; no voters:
I20260929 11:32:09.221704 12763 leader_election.cc:288] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:09.222077 12773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:09.223070 12773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c [term 1 LEADER]: Becoming Leader. State: Replica: 9b1ac4e4f7444e9bb3da87b90fdca90c, State: Running, Role: LEADER
I20260929 11:32:09.224177 12763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:09.223912 12773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b1ac4e4f7444e9bb3da87b90fdca90c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41557 } }
I20260929 11:32:09.227661 12775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b1ac4e4f7444e9bb3da87b90fdca90c. Latest consensus state: current_term: 1 leader_uuid: "9b1ac4e4f7444e9bb3da87b90fdca90c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b1ac4e4f7444e9bb3da87b90fdca90c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41557 } } }
I20260929 11:32:09.227638 12774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b1ac4e4f7444e9bb3da87b90fdca90c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b1ac4e4f7444e9bb3da87b90fdca90c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41557 } } }
I20260929 11:32:09.228315 12775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:09.228322 12774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:09.231582 12783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:09.236090 12783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:09.246924 12783 catalog_manager.cc:1347] Generated new cluster ID: 18187b4b75c74556936975b3f0aaa8f2
I20260929 11:32:09.247119 12783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:09.265105 12783 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:09.265749 12783 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:09.282583 12783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b1ac4e4f7444e9bb3da87b90fdca90c: Generated new TSK 0
I20260929 11:32:09.283934 12783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:32:09.301141 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 12696
W20260929 11:32:09.302549 12758 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35909
W20260929 11:32:09.303349 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 12696: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:32:09.303536 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 12696
2026-09-29T11:32:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1897 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:09 dist-test-slave-qk05 krb5kdc[12794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:09 dist-test-slave-qk05 krb5kdc[12794](info): set up 2 sockets
Sep 29 11:32:09 dist-test-slave-qk05 krb5kdc[12794](info): commencing operation
krb5kdc: starting...
W20260929 11:32:10.351706 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:10 dist-test-slave-qk05 krb5kdc[12794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:32:10.630128 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:10.989514 12807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:10.989876 12807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:10.990020 12807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:10.998237 12807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:10.998394 12807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:10.998473 12807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:10.998551 12807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:10.998610 12807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:11.009613 12807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:11.011065 12807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:11.012501 12807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:11.025393 12813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:11.025322 12812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:11.025620 12815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:11.026337 12807 server_base.cc:1034] running on GCE node
I20260929 11:32:11.027561 12807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:11.029551 12807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:11.030824 12807 hybrid_clock.cc:648] HybridClock initialized: now 1790681531030774 us; error 51 us; skew 500 ppm
I20260929 11:32:11.031258 12807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:11 dist-test-slave-qk05 krb5kdc[12794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:11.035099 12807 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:32:11.036970 12807 webserver.cc:460] Webserver started at http://127.0.0.1:39985/ using document root <none> and password file <none>
I20260929 11:32:11.037744 12807 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:11.037884 12807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:11.038205 12807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:11.041872 12807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "6e7819ed8648462faa72cd52e284855d"
format_stamp: "Formatted at 2026-09-29 11:32:11 on dist-test-slave-qk05"
server_key: "92d1f26cf1c3bbf8887d72e2ed44e308"
server_key_iv: "378a3eb11797f55a55c3fc692f855791"
server_key_version: "encryptionkey@0"
I20260929 11:32:11.042812 12807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "6e7819ed8648462faa72cd52e284855d"
format_stamp: "Formatted at 2026-09-29 11:32:11 on dist-test-slave-qk05"
server_key: "92d1f26cf1c3bbf8887d72e2ed44e308"
server_key_iv: "378a3eb11797f55a55c3fc692f855791"
server_key_version: "encryptionkey@0"
I20260929 11:32:11.048774 12807 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260929 11:32:11.053233 12822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:11.054881 12807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:32:11.055114 12807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "6e7819ed8648462faa72cd52e284855d"
format_stamp: "Formatted at 2026-09-29 11:32:11 on dist-test-slave-qk05"
server_key: "92d1f26cf1c3bbf8887d72e2ed44e308"
server_key_iv: "378a3eb11797f55a55c3fc692f855791"
server_key_version: "encryptionkey@0"
I20260929 11:32:11.055337 12807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:11.070746 12807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:11.075263 12807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:11.075542 12807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:11.094766 12807 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33155
I20260929 11:32:11.094784 12873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33155 every 8 connection(s)
I20260929 11:32:11.096321 12807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:11.096691 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 12807
I20260929 11:32:11.096853 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:11.097543 24164 external_mini_cluster.cc:1428] Setting key b8fbd846dbe991d2a25758c8c76ec922
I20260929 11:32:11.102051 12874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 11:32:11 dist-test-slave-qk05 krb5kdc[12794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 11:32:11.115062 12874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d: Bootstrap starting.
I20260929 11:32:11.119079 12874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:11.121934 12874 log.cc:824] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:11.123874 12877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:11.099939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:43367 (local address 127.0.0.1:33155)
0929 11:32:11.100838 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:11.100850 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:11.100899 (+ 49us) server_negotiation.cc:407] Connection header received
0929 11:32:11.102828 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:11.102854 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:11.103355 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:11.103763 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:11.105342 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:11.106789 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:11.108297 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:11.109103 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:11.109299 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:11.113985 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:11.113995 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:11.114012 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:11.114061 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:11.117237 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:11.118532 (+ 1295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:11.118540 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:11.118547 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:11.118743 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:11.119503 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:11.119507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:11.119510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:11.120108 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:11.120622 (+ 514us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:11.121951 (+ 1329us) server_negotiation.cc:299] Negotiation successful
0929 11:32:11.122361 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":430,"threads_started":1}
I20260929 11:32:11.126148 12874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d: No bootstrap required, opened a new log
I20260929 11:32:11.128751 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:11.137519 12874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7819ed8648462faa72cd52e284855d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33155 } }
I20260929 11:32:11.137959 12874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:11.138062 12874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7819ed8648462faa72cd52e284855d, State: Initialized, Role: FOLLOWER
I20260929 11:32:11.138844 12874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7819ed8648462faa72cd52e284855d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33155 } }
I20260929 11:32:11.139235 12874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:11.139429 12874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:11.139621 12874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:11.142485 12874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7819ed8648462faa72cd52e284855d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33155 } }
I20260929 11:32:11.143179 12874 leader_election.cc:302] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e7819ed8648462faa72cd52e284855d; no voters:
I20260929 11:32:11.143847 12874 leader_election.cc:288] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:11.143987 12884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:11.145150 12884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d [term 1 LEADER]: Becoming Leader. State: Replica: 6e7819ed8648462faa72cd52e284855d, State: Running, Role: LEADER
I20260929 11:32:11.145905 12884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7819ed8648462faa72cd52e284855d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33155 } }
I20260929 11:32:11.146471 12874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:11.149049 12885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e7819ed8648462faa72cd52e284855d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7819ed8648462faa72cd52e284855d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33155 } } }
I20260929 11:32:11.149497 12885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:11.152217 12877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:11.135982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:54051 (local address 127.0.0.1:33155)
0929 11:32:11.136202 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:11.136213 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:11.136653 (+ 440us) server_negotiation.cc:407] Connection header received
0929 11:32:11.136791 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:11.136796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:11.136885 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:11.137006 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:11.138771 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:11.139997 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:11.141320 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:11.142008 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:11.142158 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:11.145069 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:11.145077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:11.145083 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:11.145144 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:11.148468 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:11.149621 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:11.149627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:11.149634 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:11.149845 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:11.150411 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:11.150417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:11.150421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:11.150864 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:11.151234 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:11.151627 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 11:32:11.151838 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:32:11.154934 12886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e7819ed8648462faa72cd52e284855d. Latest consensus state: current_term: 1 leader_uuid: "6e7819ed8648462faa72cd52e284855d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e7819ed8648462faa72cd52e284855d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33155 } } }
I20260929 11:32:11.155464 12886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d [sys.catalog]: This master's current role is: LEADER
W20260929 11:32:11.163511 12900 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:32:11.163664 12900 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:32:11.163815 12889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:11.165791 12889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:11.175675 12889 catalog_manager.cc:1347] Generated new cluster ID: b131c72d197f4423b3610de3180b5cbc
I20260929 11:32:11.175817 12889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:11.191550 12889 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:11.193459 12889 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:11.219177 12889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e7819ed8648462faa72cd52e284855d: Generated new TSK 0
I20260929 11:32:11.220115 12889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:32:11.232566 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 12807
I20260929 11:32:11.234455 12869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:11.333909 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 12807
W20260929 11:32:11.349352 12792 connection.cc:535] client connection to 127.0.0.1:33155 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33155): Connection reset by peer (error 104)
2026-09-29T11:32:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2030 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:11 dist-test-slave-qk05 krb5kdc[12906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:11 dist-test-slave-qk05 krb5kdc[12906](info): set up 2 sockets
Sep 29 11:32:11 dist-test-slave-qk05 krb5kdc[12906](info): commencing operation
krb5kdc: starting...
W20260929 11:32:12.391403 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:12 dist-test-slave-qk05 krb5kdc[12906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:32:12.687415 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46605
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:13.073151 12919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:13.073861 12919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:13.074043 12919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:13.082688 12919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:13.082882 12919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:13.082970 12919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:13.083081 12919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:13.083138 12919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:13.097625 12919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:13.099242 12919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:13.100682 12919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:13.115525 12924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:13.115550 12927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:13.115496 12925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:13.116621 12919 server_base.cc:1034] running on GCE node
I20260929 11:32:13.117697 12919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:13.119498 12919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:13.120904 12919 hybrid_clock.cc:648] HybridClock initialized: now 1790681533120813 us; error 82 us; skew 500 ppm
I20260929 11:32:13.121417 12919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:13 dist-test-slave-qk05 krb5kdc[12906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:13.130939 12919 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:32:13.133175 12919 webserver.cc:460] Webserver started at http://127.0.0.1:39919/ using document root <none> and password file <none>
I20260929 11:32:13.134014 12919 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:13.134146 12919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:13.134487 12919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:13.137918 12919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "2d79b7663a3045f5a520ac40e62bf7dc"
format_stamp: "Formatted at 2026-09-29 11:32:13 on dist-test-slave-qk05"
server_key: "bb0737a6355bdc52c6f06e0dd6698c13"
server_key_iv: "8df7c6d46e3ad25e398cd9eb5ea6fcc8"
server_key_version: "encryptionkey@0"
I20260929 11:32:13.139233 12919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "2d79b7663a3045f5a520ac40e62bf7dc"
format_stamp: "Formatted at 2026-09-29 11:32:13 on dist-test-slave-qk05"
server_key: "bb0737a6355bdc52c6f06e0dd6698c13"
server_key_iv: "8df7c6d46e3ad25e398cd9eb5ea6fcc8"
server_key_version: "encryptionkey@0"
I20260929 11:32:13.146852 12919 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260929 11:32:13.152731 12934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:13.154716 12919 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260929 11:32:13.154963 12919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "2d79b7663a3045f5a520ac40e62bf7dc"
format_stamp: "Formatted at 2026-09-29 11:32:13 on dist-test-slave-qk05"
server_key: "bb0737a6355bdc52c6f06e0dd6698c13"
server_key_iv: "8df7c6d46e3ad25e398cd9eb5ea6fcc8"
server_key_version: "encryptionkey@0"
I20260929 11:32:13.155450 12919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:13.184929 12919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:13.190531 12919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:13.190972 12919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:13.211994 12919 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46605
I20260929 11:32:13.212008 12985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46605 every 8 connection(s)
I20260929 11:32:13.213479 12919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:13.217900 12986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:13.223202 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 12919
I20260929 11:32:13.223603 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:13.224210 24164 external_mini_cluster.cc:1428] Setting key 912d1d8c1f71f678ecda4427fc43a639
I20260929 11:32:13.231520 12986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc: Bootstrap starting.
I20260929 11:32:13.235502 12986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:13.236877 12986 log.cc:824] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:13.240594 12989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:13.226363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:37267 (local address 127.0.0.1:46605)
0929 11:32:13.227221 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:13.227235 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:13.227278 (+ 43us) server_negotiation.cc:407] Connection header received
0929 11:32:13.229987 (+ 2709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:13.230019 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:13.230643 (+ 624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:13.231107 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:13.231799 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:13.233195 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:13.235208 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:13.236085 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:13.236241 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:13.237823 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:13.237835 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:13.237859 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:13.238678 (+ 819us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:13.238767 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:13.238797 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:13.238952 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:13.239320 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 11:32:13.239508 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":367,"threads_started":1}
I20260929 11:32:13.240797 12986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc: No bootstrap required, opened a new log
I20260929 11:32:13.245501 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:13.252281 12986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d79b7663a3045f5a520ac40e62bf7dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46605 } }
I20260929 11:32:13.252923 12986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:13.253033 12986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d79b7663a3045f5a520ac40e62bf7dc, State: Initialized, Role: FOLLOWER
I20260929 11:32:13.253690 12986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d79b7663a3045f5a520ac40e62bf7dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46605 } }
I20260929 11:32:13.253942 12986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:13.254096 12986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:13.254281 12986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:13.257493 12986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d79b7663a3045f5a520ac40e62bf7dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46605 } }
I20260929 11:32:13.258224 12986 leader_election.cc:302] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d79b7663a3045f5a520ac40e62bf7dc; no voters:
I20260929 11:32:13.259208 12989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:13.251213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:54781 (local address 127.0.0.1:46605)
0929 11:32:13.251573 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:13.251578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:13.251859 (+ 281us) server_negotiation.cc:407] Connection header received
0929 11:32:13.252091 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:13.252096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:13.252170 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:13.252274 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:13.252933 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:13.253840 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:13.255889 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:13.256528 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:13.256654 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:13.258069 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:13.258075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:13.258078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:13.258099 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:13.258164 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:13.258183 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:13.258279 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:13.258731 (+ 452us) server_negotiation.cc:299] Negotiation successful
0929 11:32:13.258789 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 11:32:13.259246 12986 leader_election.cc:288] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:13.259383 12996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:13.260903 12996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc [term 1 LEADER]: Becoming Leader. State: Replica: 2d79b7663a3045f5a520ac40e62bf7dc, State: Running, Role: LEADER
I20260929 11:32:13.261950 12996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d79b7663a3045f5a520ac40e62bf7dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46605 } }
I20260929 11:32:13.262723 12986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:13.266225 12997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d79b7663a3045f5a520ac40e62bf7dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d79b7663a3045f5a520ac40e62bf7dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46605 } } }
I20260929 11:32:13.266834 12997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:13.267287 12998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d79b7663a3045f5a520ac40e62bf7dc. Latest consensus state: current_term: 1 leader_uuid: "2d79b7663a3045f5a520ac40e62bf7dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d79b7663a3045f5a520ac40e62bf7dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46605 } } }
I20260929 11:32:13.267694 12998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:13.267881 13002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:13.276909 13002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:13.290091 13002 catalog_manager.cc:1347] Generated new cluster ID: 9c9d21a45fec412ba917b7b1eff5b174
I20260929 11:32:13.290254 13002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:13.302371 13002 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:13.304731 13002 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:13.319396 13002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d79b7663a3045f5a520ac40e62bf7dc: Generated new TSK 0
I20260929 11:32:13.320722 13002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:32:13.332109 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 12919
W20260929 11:32:13.333580 12981 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.208:37267
W20260929 11:32:13.334468 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 12919: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:32:13.334615 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 12919
W20260929 11:32:13.349597 12904 connection.cc:535] client connection to 127.0.0.1:46605 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46605): Connection reset by peer (error 104)
2026-09-29T11:32:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2000 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:13 dist-test-slave-qk05 krb5kdc[13017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:13 dist-test-slave-qk05 krb5kdc[13017](info): set up 2 sockets
Sep 29 11:32:13 dist-test-slave-qk05 krb5kdc[13017](info): commencing operation
krb5kdc: starting...
W20260929 11:32:14.403452 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.010s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:14 dist-test-slave-qk05 krb5kdc[13017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:32:14.689924 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38221
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:15.060726 13030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:15.061071 13030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:15.061192 13030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:15.068768 13030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:15.068902 13030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:15.068962 13030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:15.069051 13030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:15.069103 13030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:15.080138 13030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:15.081614 13030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:15.082986 13030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:15.096078 13036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:15.096131 13035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:15.096684 13038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:15.097113 13030 server_base.cc:1034] running on GCE node
I20260929 11:32:15.097995 13030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:15.099685 13030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:15.100965 13030 hybrid_clock.cc:648] HybridClock initialized: now 1790681535100906 us; error 57 us; skew 500 ppm
I20260929 11:32:15.101404 13030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:15 dist-test-slave-qk05 krb5kdc[13017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:15.104908 13030 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:32:15.106544 13030 webserver.cc:460] Webserver started at http://127.0.0.1:45915/ using document root <none> and password file <none>
I20260929 11:32:15.107357 13030 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:15.107478 13030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:15.107770 13030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:15.110998 13030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "abf661dc346a4eb2b0802ca1c72f1bda"
format_stamp: "Formatted at 2026-09-29 11:32:15 on dist-test-slave-qk05"
server_key: "1fd0c9d046a30223a4f6de25668177ce"
server_key_iv: "9fd7ab9ba0f3de3a0158c9f3803f35a3"
server_key_version: "encryptionkey@0"
I20260929 11:32:15.111909 13030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "abf661dc346a4eb2b0802ca1c72f1bda"
format_stamp: "Formatted at 2026-09-29 11:32:15 on dist-test-slave-qk05"
server_key: "1fd0c9d046a30223a4f6de25668177ce"
server_key_iv: "9fd7ab9ba0f3de3a0158c9f3803f35a3"
server_key_version: "encryptionkey@0"
I20260929 11:32:15.117805 13030 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260929 11:32:15.122062 13045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:15.123752 13030 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 11:32:15.123955 13030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "abf661dc346a4eb2b0802ca1c72f1bda"
format_stamp: "Formatted at 2026-09-29 11:32:15 on dist-test-slave-qk05"
server_key: "1fd0c9d046a30223a4f6de25668177ce"
server_key_iv: "9fd7ab9ba0f3de3a0158c9f3803f35a3"
server_key_version: "encryptionkey@0"
I20260929 11:32:15.124181 13030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:15.131453 13030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:15.136291 13030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:15.136590 13030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:15.158377 13030 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38221
I20260929 11:32:15.158408 13096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38221 every 8 connection(s)
I20260929 11:32:15.159962 13030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:15.164906 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:15.166998 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 13030
I20260929 11:32:15.167232 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:15.167785 24164 external_mini_cluster.cc:1428] Setting key 35fae3fa6c8928098edcf40f4cab5de4
I20260929 11:32:15.176398 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:15.169753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:49109 (local address 127.0.0.1:38221)
0929 11:32:15.170679 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:15.170694 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:15.170739 (+ 45us) server_negotiation.cc:407] Connection header received
0929 11:32:15.172506 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:15.172538 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:15.173028 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:15.173554 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:15.174632 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:15.174640 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:15.174655 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:15.174670 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:15.174731 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:15.174753 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:15.174935 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:15.175077 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 11:32:15.175324 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":401,"threads_started":1}
I20260929 11:32:15.179728 13097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda: Bootstrap starting.
I20260929 11:32:15.180790 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:15.184634 13097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:15.186434 13097 log.cc:824] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:15.189886 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:15.187367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:36459 (local address 127.0.0.1:38221)
0929 11:32:15.187526 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:15.187531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:15.187879 (+ 348us) server_negotiation.cc:407] Connection header received
0929 11:32:15.188075 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:15.188080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:15.188139 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:15.188247 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:15.189138 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:15.189144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:15.189149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:15.189168 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:15.189220 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:15.189240 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:15.189376 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:15.189507 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 11:32:15.189576 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 11:32:15.190284 13097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda: No bootstrap required, opened a new log
I20260929 11:32:15.199121 13097 raft_consensus.cc:348] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf661dc346a4eb2b0802ca1c72f1bda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38221 } }
I20260929 11:32:15.199739 13097 raft_consensus.cc:374] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:15.199857 13097 raft_consensus.cc:729] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf661dc346a4eb2b0802ca1c72f1bda, State: Initialized, Role: FOLLOWER
I20260929 11:32:15.200748 13097 consensus_queue.cc:260] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf661dc346a4eb2b0802ca1c72f1bda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38221 } }
I20260929 11:32:15.201288 13097 raft_consensus.cc:388] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:15.201490 13097 raft_consensus.cc:482] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:15.201656 13097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:15.204823 13097 raft_consensus.cc:504] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf661dc346a4eb2b0802ca1c72f1bda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38221 } }
I20260929 11:32:15.205613 13097 leader_election.cc:302] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abf661dc346a4eb2b0802ca1c72f1bda; no voters:
I20260929 11:32:15.206262 13097 leader_election.cc:288] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:15.206389 13107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:15.207288 13107 raft_consensus.cc:686] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda [term 1 LEADER]: Becoming Leader. State: Replica: abf661dc346a4eb2b0802ca1c72f1bda, State: Running, Role: LEADER
I20260929 11:32:15.207942 13107 consensus_queue.cc:237] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf661dc346a4eb2b0802ca1c72f1bda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38221 } }
I20260929 11:32:15.208443 13097 sys_catalog.cc:564] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:15.211179 13109 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda [sys.catalog]: SysCatalogTable state changed. Reason: New leader abf661dc346a4eb2b0802ca1c72f1bda. Latest consensus state: current_term: 1 leader_uuid: "abf661dc346a4eb2b0802ca1c72f1bda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf661dc346a4eb2b0802ca1c72f1bda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38221 } } }
I20260929 11:32:15.211375 13108 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abf661dc346a4eb2b0802ca1c72f1bda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf661dc346a4eb2b0802ca1c72f1bda" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38221 } } }
I20260929 11:32:15.211717 13109 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:15.211717 13108 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:15.214409 13115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:15.220515 13115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:15.231737 13115 catalog_manager.cc:1347] Generated new cluster ID: 0e00ddb022834bd190357179515f770a
I20260929 11:32:15.231879 13115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:15.246191 13115 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:15.246713 13115 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:15.254828 13115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abf661dc346a4eb2b0802ca1c72f1bda: Generated new TSK 0
I20260929 11:32:15.255857 13115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:32:15.275007 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 13030
W20260929 11:32:15.276260 13092 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.208:49109
W20260929 11:32:15.277021 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 13030: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:32:15.277190 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 13030
2026-09-29T11:32:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1942 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:15 dist-test-slave-qk05 krb5kdc[13128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:15 dist-test-slave-qk05 krb5kdc[13128](info): set up 2 sockets
Sep 29 11:32:15 dist-test-slave-qk05 krb5kdc[13128](info): commencing operation
krb5kdc: starting...
W20260929 11:32:16.343670 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:16 dist-test-slave-qk05 krb5kdc[13128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:32:16.612074 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42899
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:16.966143 13141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:16.966470 13141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:16.966602 13141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:16.975411 13141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:16.975565 13141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:16.975646 13141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:16.975726 13141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:16.975800 13141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:16.986961 13141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42899
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:16.988402 13141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:16.989854 13141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:17.002480 13147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:17.002286 13149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:17.002527 13146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:17.003830 13141 server_base.cc:1034] running on GCE node
I20260929 11:32:17.005023 13141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:17.006759 13141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:17.008028 13141 hybrid_clock.cc:648] HybridClock initialized: now 1790681537007963 us; error 67 us; skew 500 ppm
I20260929 11:32:17.008492 13141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:17 dist-test-slave-qk05 krb5kdc[13128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:17.012043 13141 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:32:17.013790 13141 webserver.cc:460] Webserver started at http://127.0.0.1:39839/ using document root <none> and password file <none>
I20260929 11:32:17.014709 13141 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:17.014878 13141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:17.015256 13141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:17.018651 13141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "218668b5eef74747970ec848f049e5bc"
format_stamp: "Formatted at 2026-09-29 11:32:17 on dist-test-slave-qk05"
server_key: "549be8def4022c03a055d607248ecafd"
server_key_iv: "8dde4eb0c28ecaaae240eb6cf8ae12fb"
server_key_version: "encryptionkey@0"
I20260929 11:32:17.019991 13141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "218668b5eef74747970ec848f049e5bc"
format_stamp: "Formatted at 2026-09-29 11:32:17 on dist-test-slave-qk05"
server_key: "549be8def4022c03a055d607248ecafd"
server_key_iv: "8dde4eb0c28ecaaae240eb6cf8ae12fb"
server_key_version: "encryptionkey@0"
I20260929 11:32:17.028797 13141 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260929 11:32:17.033162 13156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:17.034761 13141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 11:32:17.034981 13141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "218668b5eef74747970ec848f049e5bc"
format_stamp: "Formatted at 2026-09-29 11:32:17 on dist-test-slave-qk05"
server_key: "549be8def4022c03a055d607248ecafd"
server_key_iv: "8dde4eb0c28ecaaae240eb6cf8ae12fb"
server_key_version: "encryptionkey@0"
I20260929 11:32:17.035287 13141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:17.053666 13141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:17.058501 13141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:17.058815 13141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:17.081306 13141 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42899
I20260929 11:32:17.081372 13207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42899 every 8 connection(s)
I20260929 11:32:17.082883 13141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:17.087467 13208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:17.091288 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 13141
I20260929 11:32:17.091490 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:17.092033 24164 external_mini_cluster.cc:1428] Setting key 7eb1c2f4de2806298a7ffc2d0ea4e0d7
I20260929 11:32:17.101078 13208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc: Bootstrap starting.
I20260929 11:32:17.104668 13208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:17.105855 13208 log.cc:824] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc: Log is configured to *not* fsync() on all Append() calls
Sep 29 11:32:17 dist-test-slave-qk05 krb5kdc[13128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 11:32:17.109568 13208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc: No bootstrap required, opened a new log
I20260929 11:32:17.121055 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:17.094177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:38531 (local address 127.0.0.1:42899)
0929 11:32:17.095184 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:17.095200 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:17.095250 (+ 50us) server_negotiation.cc:407] Connection header received
0929 11:32:17.097084 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:17.097114 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:17.097619 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:17.098076 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:17.099666 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:17.101359 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:17.102783 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:17.104311 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:17.104712 (+ 401us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:17.109836 (+ 5124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:17.109849 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:17.109873 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:17.109926 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:17.114697 (+ 4771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:17.116006 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:17.116017 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:17.116028 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:17.116297 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:17.116850 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:17.116855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:17.116862 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:17.117714 (+ 852us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:17.118097 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:17.119047 (+ 950us) server_negotiation.cc:299] Negotiation successful
0929 11:32:17.119507 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":445,"threads_started":1}
I20260929 11:32:17.121114 13208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218668b5eef74747970ec848f049e5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } }
I20260929 11:32:17.121585 13208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:17.121699 13208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218668b5eef74747970ec848f049e5bc, State: Initialized, Role: FOLLOWER
I20260929 11:32:17.122434 13208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218668b5eef74747970ec848f049e5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } }
I20260929 11:32:17.122738 13208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:17.122886 13208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:17.123068 13208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:17.126050 13208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218668b5eef74747970ec848f049e5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } }
I20260929 11:32:17.126528 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:17.126717 13208 leader_election.cc:302] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 218668b5eef74747970ec848f049e5bc; no voters:
I20260929 11:32:17.127586 13208 leader_election.cc:288] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:17.127768 13213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:17.129830 13208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:17.136191 13213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc [term 1 LEADER]: Becoming Leader. State: Replica: 218668b5eef74747970ec848f049e5bc, State: Running, Role: LEADER
I20260929 11:32:17.137286 13213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218668b5eef74747970ec848f049e5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } }
I20260929 11:32:17.141028 13214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "218668b5eef74747970ec848f049e5bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218668b5eef74747970ec848f049e5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } } }
I20260929 11:32:17.141472 13214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:17.142004 13215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 218668b5eef74747970ec848f049e5bc. Latest consensus state: current_term: 1 leader_uuid: "218668b5eef74747970ec848f049e5bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "218668b5eef74747970ec848f049e5bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } } }
I20260929 11:32:17.142323 13215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:17.150104 13231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:17.156002 13231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:17.167052 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:17.139380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:32991 (local address 127.0.0.1:42899)
0929 11:32:17.139609 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:17.139616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:17.148051 (+ 8435us) server_negotiation.cc:407] Connection header received
0929 11:32:17.148406 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:17.148414 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:17.148528 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:17.148615 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:17.151855 (+ 3240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:17.152753 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:17.154166 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:17.154653 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:17.154784 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:17.158479 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:17.158486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:17.158491 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:17.158541 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:17.162806 (+ 4265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:17.164163 (+ 1357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:17.164170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:17.164173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:17.164592 (+ 419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:17.165251 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:17.165255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:17.165257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:17.165604 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:17.165902 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:17.166450 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 11:32:17.166661 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 11:32:17.167939 13231 catalog_manager.cc:1347] Generated new cluster ID: c1eead77d49848b292c48b8c89ef26b3
I20260929 11:32:17.168061 13231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:17.190208 13231 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:17.192732 13231 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:17.208007 13231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 218668b5eef74747970ec848f049e5bc: Generated new TSK 0
I20260929 11:32:17.209086 13231 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:32:17.222096 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 13141
I20260929 11:32:17.223640 13203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:17.320667 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 13141
W20260929 11:32:17.336246 13126 connection.cc:535] client connection to 127.0.0.1:42899 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42899): Connection reset by peer (error 104)
2026-09-29T11:32:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2043 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:17 dist-test-slave-qk05 krb5kdc[13241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:17 dist-test-slave-qk05 krb5kdc[13241](info): set up 2 sockets
Sep 29 11:32:17 dist-test-slave-qk05 krb5kdc[13241](info): commencing operation
krb5kdc: starting...
W20260929 11:32:18.371786 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:18 dist-test-slave-qk05 krb5kdc[13241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:32:18.651726 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:19.024295 13254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:19.024624 13254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:19.024722 13254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:19.032781 13254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:19.032943 13254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:19.032994 13254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:19.033056 13254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:19.033100 13254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:19.044646 13254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40173
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:19.046051 13254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:19.047510 13254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:19.060088 13259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:19.060175 13260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:19.060108 13262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:19.060946 13254 server_base.cc:1034] running on GCE node
I20260929 11:32:19.061991 13254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:19.063710 13254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:19.064988 13254 hybrid_clock.cc:648] HybridClock initialized: now 1790681539064928 us; error 49 us; skew 500 ppm
I20260929 11:32:19.065743 13254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:19 dist-test-slave-qk05 krb5kdc[13241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:19.077145 13254 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:32:19.078948 13254 webserver.cc:460] Webserver started at http://127.0.0.1:43569/ using document root <none> and password file <none>
I20260929 11:32:19.079788 13254 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:19.079916 13254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:19.080227 13254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:19.083595 13254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "f43318c786d846a8b448bab5e5205e58"
format_stamp: "Formatted at 2026-09-29 11:32:19 on dist-test-slave-qk05"
server_key: "b44a4702bfbc20b09fc844e55fc03a51"
server_key_iv: "a8ce3053da43b714cf880a1bc3d9bde2"
server_key_version: "encryptionkey@0"
I20260929 11:32:19.084461 13254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "f43318c786d846a8b448bab5e5205e58"
format_stamp: "Formatted at 2026-09-29 11:32:19 on dist-test-slave-qk05"
server_key: "b44a4702bfbc20b09fc844e55fc03a51"
server_key_iv: "a8ce3053da43b714cf880a1bc3d9bde2"
server_key_version: "encryptionkey@0"
I20260929 11:32:19.090687 13254 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 11:32:19.094899 13269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:19.096997 13254 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:32:19.097236 13254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "f43318c786d846a8b448bab5e5205e58"
format_stamp: "Formatted at 2026-09-29 11:32:19 on dist-test-slave-qk05"
server_key: "b44a4702bfbc20b09fc844e55fc03a51"
server_key_iv: "a8ce3053da43b714cf880a1bc3d9bde2"
server_key_version: "encryptionkey@0"
I20260929 11:32:19.097560 13254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:19.133231 13254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:19.139366 13254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:19.139916 13254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:19.161571 13254 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40173
I20260929 11:32:19.161577 13320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40173 every 8 connection(s)
I20260929 11:32:19.164031 13254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:19.164513 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 13254
I20260929 11:32:19.164705 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:19.165278 24164 external_mini_cluster.cc:1428] Setting key 9e606d2895960a9ab5e26ecf75ea107b
I20260929 11:32:19.169775 13321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:19.186136 13321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58: Bootstrap starting.
I20260929 11:32:19.186782 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:19.171411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:46019 (local address 127.0.0.1:40173)
0929 11:32:19.172298 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:19.172311 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:19.172356 (+ 45us) server_negotiation.cc:407] Connection header received
0929 11:32:19.174971 (+ 2615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:19.175002 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:19.175612 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:19.176034 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:19.177118 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:19.178611 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:19.180622 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:19.181431 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:19.181593 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:19.183527 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:19.183553 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:19.183586 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:19.184565 (+ 979us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:19.184674 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:19.184710 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:19.184874 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:19.185220 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 11:32:19.185471 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":389,"threads_started":1}
I20260929 11:32:19.190044 13321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:19.191452 13321 log.cc:824] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:19.191730 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:19.194868 13321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58: No bootstrap required, opened a new log
I20260929 11:32:19.205478 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:19.196838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:44047 (local address 127.0.0.1:40173)
0929 11:32:19.197053 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:19.197059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:19.197415 (+ 356us) server_negotiation.cc:407] Connection header received
0929 11:32:19.197683 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:19.197688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:19.197769 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:19.197878 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:19.198516 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:19.199641 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:19.201817 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:19.202408 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:19.202558 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:19.204352 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:19.204371 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:19.204376 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:19.204409 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:19.204468 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:19.204494 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:19.204616 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:19.204892 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 11:32:19.204956 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 11:32:19.205154 13321 raft_consensus.cc:348] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43318c786d846a8b448bab5e5205e58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40173 } }
I20260929 11:32:19.205694 13321 raft_consensus.cc:374] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:19.205830 13321 raft_consensus.cc:729] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f43318c786d846a8b448bab5e5205e58, State: Initialized, Role: FOLLOWER
I20260929 11:32:19.206538 13321 consensus_queue.cc:260] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43318c786d846a8b448bab5e5205e58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40173 } }
I20260929 11:32:19.206840 13321 raft_consensus.cc:388] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:19.206995 13321 raft_consensus.cc:482] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:19.207242 13321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:19.210075 13321 raft_consensus.cc:504] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43318c786d846a8b448bab5e5205e58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40173 } }
I20260929 11:32:19.210981 13321 leader_election.cc:302] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f43318c786d846a8b448bab5e5205e58; no voters:
I20260929 11:32:19.211815 13321 leader_election.cc:288] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:19.211984 13331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:19.212822 13331 raft_consensus.cc:686] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58 [term 1 LEADER]: Becoming Leader. State: Replica: f43318c786d846a8b448bab5e5205e58, State: Running, Role: LEADER
I20260929 11:32:19.213598 13331 consensus_queue.cc:237] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43318c786d846a8b448bab5e5205e58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40173 } }
I20260929 11:32:19.214217 13321 sys_catalog.cc:564] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:19.216859 13333 sys_catalog.cc:455] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f43318c786d846a8b448bab5e5205e58. Latest consensus state: current_term: 1 leader_uuid: "f43318c786d846a8b448bab5e5205e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43318c786d846a8b448bab5e5205e58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40173 } } }
I20260929 11:32:19.216920 13332 sys_catalog.cc:455] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f43318c786d846a8b448bab5e5205e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f43318c786d846a8b448bab5e5205e58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40173 } } }
I20260929 11:32:19.217360 13332 sys_catalog.cc:458] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:19.217358 13333 sys_catalog.cc:458] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:19.219262 13338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:19.225421 13338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:19.238741 13338 catalog_manager.cc:1347] Generated new cluster ID: c0ac885ae34d49aa887dbc6267b71cc2
I20260929 11:32:19.238900 13338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:19.267943 13338 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:19.270026 13338 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:19.279420 13338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f43318c786d846a8b448bab5e5205e58: Generated new TSK 0
I20260929 11:32:19.280640 13338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:32:19.295328 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 13254
W20260929 11:32:19.296792 13316 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.208:46019
W20260929 11:32:19.297967 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 13254: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:32:19.298192 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 13254
W20260929 11:32:19.313994 13239 connection.cc:535] client connection to 127.0.0.1:40173 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40173): Connection reset by peer (error 104)
2026-09-29T11:32:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1977 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:19 dist-test-slave-qk05 krb5kdc[13352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:19 dist-test-slave-qk05 krb5kdc[13352](info): set up 2 sockets
Sep 29 11:32:19 dist-test-slave-qk05 krb5kdc[13352](info): commencing operation
krb5kdc: starting...
W20260929 11:32:20.356537 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:20 dist-test-slave-qk05 krb5kdc[13352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 11:32:20.648032 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:21.026389 13365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:21.026714 13365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:21.026815 13365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:21.035228 13365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:21.035388 13365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:21.035449 13365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:21.035522 13365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:21.035583 13365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:21.046178 13365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34989
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:21.047613 13365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:21.048979 13365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:21.063119 13371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:21.063174 13370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:21.063153 13373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:21.064087 13365 server_base.cc:1034] running on GCE node
I20260929 11:32:21.065078 13365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:21.066874 13365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:21.068126 13365 hybrid_clock.cc:648] HybridClock initialized: now 1790681541068082 us; error 57 us; skew 500 ppm
I20260929 11:32:21.068641 13365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:21 dist-test-slave-qk05 krb5kdc[13352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:21.072228 13365 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 11:32:21.073982 13365 webserver.cc:460] Webserver started at http://127.0.0.1:35843/ using document root <none> and password file <none>
I20260929 11:32:21.074761 13365 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:21.074899 13365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:21.075277 13365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:21.078806 13365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "b98dad42435a4a17995bc4f941aaa9aa"
format_stamp: "Formatted at 2026-09-29 11:32:21 on dist-test-slave-qk05"
server_key: "7c26b54221995296de1976cf41eaa36d"
server_key_iv: "182e33f0427211e315bffe6a4779a8a7"
server_key_version: "encryptionkey@0"
I20260929 11:32:21.079819 13365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "b98dad42435a4a17995bc4f941aaa9aa"
format_stamp: "Formatted at 2026-09-29 11:32:21 on dist-test-slave-qk05"
server_key: "7c26b54221995296de1976cf41eaa36d"
server_key_iv: "182e33f0427211e315bffe6a4779a8a7"
server_key_version: "encryptionkey@0"
I20260929 11:32:21.086205 13365 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260929 11:32:21.090823 13380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:21.092597 13365 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260929 11:32:21.092846 13365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "b98dad42435a4a17995bc4f941aaa9aa"
format_stamp: "Formatted at 2026-09-29 11:32:21 on dist-test-slave-qk05"
server_key: "7c26b54221995296de1976cf41eaa36d"
server_key_iv: "182e33f0427211e315bffe6a4779a8a7"
server_key_version: "encryptionkey@0"
I20260929 11:32:21.093101 13365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:21.100085 13365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:21.105288 13365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:21.105655 13365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:21.127440 13365 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34989
I20260929 11:32:21.127478 13431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34989 every 8 connection(s)
I20260929 11:32:21.129061 13365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:21.133263 13432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:21.138489 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 13365
I20260929 11:32:21.138701 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:21.139178 24164 external_mini_cluster.cc:1428] Setting key 560c9f680bb378bcf4335ce56bc08947
I20260929 11:32:21.147979 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:21.141286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:33127 (local address 127.0.0.1:34989)
0929 11:32:21.142145 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:21.142158 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:21.142202 (+ 44us) server_negotiation.cc:407] Connection header received
0929 11:32:21.143993 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:21.144023 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:21.144482 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:21.144885 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:21.146207 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:21.146214 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:21.146233 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:21.146250 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:21.146317 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:21.146340 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:21.146538 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:21.146648 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 11:32:21.146881 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":349,"threads_started":1}
I20260929 11:32:21.148777 13432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa: Bootstrap starting.
I20260929 11:32:21.152348 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:21.152379 13432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:21.153565 13432 log.cc:824] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:21.157101 13432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa: No bootstrap required, opened a new log
I20260929 11:32:21.159953 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:21.156954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.208:40817 (local address 127.0.0.1:34989)
0929 11:32:21.157175 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:21.157182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:21.157762 (+ 580us) server_negotiation.cc:407] Connection header received
0929 11:32:21.157940 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:21.157944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:21.158001 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:21.158098 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:21.159149 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:21.159155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:21.159160 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:21.159179 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:21.159236 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:21.159350 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:21.159502 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:21.159623 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 11:32:21.159691 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 11:32:21.167066 13432 raft_consensus.cc:348] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98dad42435a4a17995bc4f941aaa9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34989 } }
I20260929 11:32:21.167557 13432 raft_consensus.cc:374] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:21.167665 13432 raft_consensus.cc:729] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98dad42435a4a17995bc4f941aaa9aa, State: Initialized, Role: FOLLOWER
I20260929 11:32:21.168413 13432 consensus_queue.cc:260] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98dad42435a4a17995bc4f941aaa9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34989 } }
I20260929 11:32:21.168807 13432 raft_consensus.cc:388] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:21.168958 13432 raft_consensus.cc:482] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:21.169129 13432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:21.171582 13432 raft_consensus.cc:504] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98dad42435a4a17995bc4f941aaa9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34989 } }
I20260929 11:32:21.172209 13432 leader_election.cc:302] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b98dad42435a4a17995bc4f941aaa9aa; no voters:
I20260929 11:32:21.172932 13432 leader_election.cc:288] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:21.173099 13442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:21.175751 13432 sys_catalog.cc:564] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:21.177043 13442 raft_consensus.cc:686] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa [term 1 LEADER]: Becoming Leader. State: Replica: b98dad42435a4a17995bc4f941aaa9aa, State: Running, Role: LEADER
I20260929 11:32:21.177861 13442 consensus_queue.cc:237] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98dad42435a4a17995bc4f941aaa9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34989 } }
I20260929 11:32:21.181521 13443 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b98dad42435a4a17995bc4f941aaa9aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98dad42435a4a17995bc4f941aaa9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34989 } } }
I20260929 11:32:21.181653 13444 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader b98dad42435a4a17995bc4f941aaa9aa. Latest consensus state: current_term: 1 leader_uuid: "b98dad42435a4a17995bc4f941aaa9aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98dad42435a4a17995bc4f941aaa9aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34989 } } }
I20260929 11:32:21.182130 13444 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:21.182132 13443 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:21.184753 13452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:21.191207 13452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:21.203305 13452 catalog_manager.cc:1347] Generated new cluster ID: a174ad5cb5c5499ea354ad25a5927913
I20260929 11:32:21.203466 13452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:21.220506 13452 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:21.221071 13452 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:21.230639 13452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b98dad42435a4a17995bc4f941aaa9aa: Generated new TSK 0
I20260929 11:32:21.231642 13452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:32:21.237601 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 13365
W20260929 11:32:21.239068 13427 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.208:33127
W20260929 11:32:21.239861 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 13365: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:32:21.240022 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 13365
2026-09-29T11:32:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1941 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:21 dist-test-slave-qk05 krb5kdc[13463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:21 dist-test-slave-qk05 krb5kdc[13463](info): set up 2 sockets
Sep 29 11:32:21 dist-test-slave-qk05 krb5kdc[13463](info): commencing operation
krb5kdc: starting...
W20260929 11:32:22.307725 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:22 dist-test-slave-qk05 krb5kdc[13463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:32:22.579205 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:34201
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:34201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:22.949208 13476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:22.949929 13476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:22.950140 13476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:22.958649 13476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:22.958817 13476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:22.958866 13476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:22.958921 13476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:22.958961 13476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:22.970398 13476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:34201
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:34201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:22.971874 13476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:22.973291 13476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:22.985827 13484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:22.985867 13482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:22.985888 13481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:22.987406 13476 server_base.cc:1034] running on GCE node
I20260929 11:32:22.988343 13476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:22.990032 13476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:22.993501 13476 hybrid_clock.cc:648] HybridClock initialized: now 1790681542993407 us; error 149 us; skew 500 ppm
I20260929 11:32:22.993994 13476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:22 dist-test-slave-qk05 krb5kdc[13463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681542, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:22.997948 13476 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:32:22.999828 13476 webserver.cc:460] Webserver started at http://127.23.153.62:39539/ using document root <none> and password file <none>
I20260929 11:32:23.000583 13476 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:23.000706 13476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:23.001041 13476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:23.004312 13476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "1978a24565c749a6bf3afdff8aa80e53"
format_stamp: "Formatted at 2026-09-29 11:32:22 on dist-test-slave-qk05"
server_key: "45c0c1cd61c291db8d2d32ab13eb53cb"
server_key_iv: "9cb82cb4582722f34955034ddcfc1bf9"
server_key_version: "encryptionkey@0"
I20260929 11:32:23.005173 13476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "1978a24565c749a6bf3afdff8aa80e53"
format_stamp: "Formatted at 2026-09-29 11:32:22 on dist-test-slave-qk05"
server_key: "45c0c1cd61c291db8d2d32ab13eb53cb"
server_key_iv: "9cb82cb4582722f34955034ddcfc1bf9"
server_key_version: "encryptionkey@0"
I20260929 11:32:23.011999 13476 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 11:32:23.016075 13491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:23.017706 13476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 11:32:23.017905 13476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "1978a24565c749a6bf3afdff8aa80e53"
format_stamp: "Formatted at 2026-09-29 11:32:22 on dist-test-slave-qk05"
server_key: "45c0c1cd61c291db8d2d32ab13eb53cb"
server_key_iv: "9cb82cb4582722f34955034ddcfc1bf9"
server_key_version: "encryptionkey@0"
I20260929 11:32:23.018123 13476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:23.034969 13476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:23.039732 13476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:23.040024 13476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:23.059259 13476 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:34201
I20260929 11:32:23.059278 13542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:34201 every 8 connection(s)
I20260929 11:32:23.060734 13476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:23.064975 13543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:23.066711 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 13476
I20260929 11:32:23.066897 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:23.067480 24164 external_mini_cluster.cc:1428] Setting key 6feaebe74be8bbf1a707188139c179e1
I20260929 11:32:23.079181 13543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53: Bootstrap starting.
Sep 29 11:32:23 dist-test-slave-qk05 krb5kdc[13463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:32:23.082597 13543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:23.084033 13543 log.cc:824] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:23.087587 13543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53: No bootstrap required, opened a new log
I20260929 11:32:23.091885 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:23.069447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32971 (local address 127.23.153.62:34201)
0929 11:32:23.070326 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:23.070343 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:23.070392 (+ 49us) server_negotiation.cc:407] Connection header received
0929 11:32:23.072243 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:23.072273 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:23.072814 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:23.073300 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:23.074912 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:23.076411 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:23.077798 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:23.078586 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:23.078774 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:23.083418 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:23.083427 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:23.083444 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:23.083490 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:23.086809 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:23.088031 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:23.088041 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:23.088051 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:23.088259 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:23.088798 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:23.088802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:23.088804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:23.089396 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:23.089705 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:23.090213 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 11:32:23.090588 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":374,"threads_started":1}
I20260929 11:32:23.096971 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:23.097610 13543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1978a24565c749a6bf3afdff8aa80e53" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34201 } }
I20260929 11:32:23.098138 13543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:23.098239 13543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1978a24565c749a6bf3afdff8aa80e53, State: Initialized, Role: FOLLOWER
I20260929 11:32:23.098955 13543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1978a24565c749a6bf3afdff8aa80e53" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34201 } }
I20260929 11:32:23.099316 13543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:23.099483 13543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:23.099668 13543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:23.102918 13543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1978a24565c749a6bf3afdff8aa80e53" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34201 } }
I20260929 11:32:23.103669 13543 leader_election.cc:302] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1978a24565c749a6bf3afdff8aa80e53; no voters:
I20260929 11:32:23.104374 13543 leader_election.cc:288] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:23.104477 13553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:23.105818 13553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53 [term 1 LEADER]: Becoming Leader. State: Replica: 1978a24565c749a6bf3afdff8aa80e53, State: Running, Role: LEADER
I20260929 11:32:23.106449 13553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1978a24565c749a6bf3afdff8aa80e53" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34201 } }
I20260929 11:32:23.106966 13543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:23.109354 13554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1978a24565c749a6bf3afdff8aa80e53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1978a24565c749a6bf3afdff8aa80e53" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34201 } } }
I20260929 11:32:23.109808 13554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:23.112085 13555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1978a24565c749a6bf3afdff8aa80e53. Latest consensus state: current_term: 1 leader_uuid: "1978a24565c749a6bf3afdff8aa80e53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1978a24565c749a6bf3afdff8aa80e53" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 34201 } } }
I20260929 11:32:23.112480 13555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:23.115478 13558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:23.122350 13558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:23.124531 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:23.102419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53743 (local address 127.23.153.62:34201)
0929 11:32:23.102627 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:23.102633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:23.102959 (+ 326us) server_negotiation.cc:407] Connection header received
0929 11:32:23.103184 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:23.103190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:23.103269 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:23.103355 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:23.105308 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:23.106507 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:23.108629 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:23.109808 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:23.109965 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:23.114861 (+ 4896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:23.114871 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:23.114877 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:23.114936 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:23.119601 (+ 4665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:23.121002 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:23.121009 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:23.121014 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:23.121216 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:23.121906 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:23.121913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:23.121918 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:23.122596 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:23.122912 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:23.123797 (+ 885us) server_negotiation.cc:299] Negotiation successful
0929 11:32:23.124079 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 11:32:23.136472 13558 catalog_manager.cc:1347] Generated new cluster ID: bf129d18078e48c6b498e7efb38d50a3
I20260929 11:32:23.136627 13558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:23.163954 13558 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:23.166309 13558 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:23.174793 13558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1978a24565c749a6bf3afdff8aa80e53: Generated new TSK 0
I20260929 11:32:23.176105 13558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:32:23.187338 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 13476
I20260929 11:32:23.188910 13538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:23.292441 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 13476
W20260929 11:32:23.309535 13461 connection.cc:535] client connection to 127.23.153.62:34201 recv error: Network error: failed to read from TLS socket (remote: 127.23.153.62:34201): Connection reset by peer (error 104)
2026-09-29T11:32:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2053 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:23 dist-test-slave-qk05 krb5kdc[13575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:23 dist-test-slave-qk05 krb5kdc[13575](info): set up 2 sockets
Sep 29 11:32:23 dist-test-slave-qk05 krb5kdc[13575](info): commencing operation
krb5kdc: starting...
W20260929 11:32:24.343540 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.009s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:24 dist-test-slave-qk05 krb5kdc[13575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:32:24.603675 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:42309
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:43895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:42309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:24.967162 13588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:24.967492 13588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:24.967628 13588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:24.975193 13588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:24.975327 13588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:24.975401 13588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:24.975485 13588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:24.975554 13588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:24.986029 13588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:43895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:42309
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:42309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:24.987672 13588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:24.989264 13588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:25.001828 13594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:25.001921 13596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:25.001886 13593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:25.002735 13588 server_base.cc:1034] running on GCE node
I20260929 11:32:25.003700 13588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:25.005517 13588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:25.006827 13588 hybrid_clock.cc:648] HybridClock initialized: now 1790681545006748 us; error 58 us; skew 500 ppm
I20260929 11:32:25.007475 13588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:25 dist-test-slave-qk05 krb5kdc[13575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681545, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:25.016112 13588 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:32:25.018430 13588 webserver.cc:460] Webserver started at http://127.23.153.62:43065/ using document root <none> and password file <none>
I20260929 11:32:25.019824 13588 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:25.020010 13588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:25.020363 13588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:25.023880 13588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "91a5ed2d733c415194a763ee1286e5d0"
format_stamp: "Formatted at 2026-09-29 11:32:25 on dist-test-slave-qk05"
server_key: "1082b2b41f69612fbdb784a26d0c93ec"
server_key_iv: "11624feaa2cf4538c8391ec2d8f22caa"
server_key_version: "encryptionkey@0"
I20260929 11:32:25.024807 13588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "91a5ed2d733c415194a763ee1286e5d0"
format_stamp: "Formatted at 2026-09-29 11:32:25 on dist-test-slave-qk05"
server_key: "1082b2b41f69612fbdb784a26d0c93ec"
server_key_iv: "11624feaa2cf4538c8391ec2d8f22caa"
server_key_version: "encryptionkey@0"
I20260929 11:32:25.032002 13588 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260929 11:32:25.036298 13603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:25.037978 13588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260929 11:32:25.038206 13588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "91a5ed2d733c415194a763ee1286e5d0"
format_stamp: "Formatted at 2026-09-29 11:32:25 on dist-test-slave-qk05"
server_key: "1082b2b41f69612fbdb784a26d0c93ec"
server_key_iv: "11624feaa2cf4538c8391ec2d8f22caa"
server_key_version: "encryptionkey@0"
I20260929 11:32:25.038424 13588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:25.057686 13588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:25.063126 13588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:25.063405 13588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:25.083072 13588 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:42309
I20260929 11:32:25.083122 13654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:42309 every 8 connection(s)
I20260929 11:32:25.084623 13588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:25.089262 13655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:25.091471 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 13588
I20260929 11:32:25.091653 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:25.092083 24164 external_mini_cluster.cc:1428] Setting key 3aa8989e35434b05979dae884726b9c6
I20260929 11:32:25.104651 13655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0: Bootstrap starting.
I20260929 11:32:25.109210 13655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:25.110298 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:25.094241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54329 (local address 127.23.153.62:42309)
0929 11:32:25.095234 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:25.095253 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:25.095314 (+ 61us) server_negotiation.cc:407] Connection header received
0929 11:32:25.097767 (+ 2453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:25.097803 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:25.098416 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:25.098934 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:25.099591 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:25.101804 (+ 2213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:25.103547 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:25.104850 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:25.105207 (+ 357us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:25.106623 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:25.106641 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:25.106676 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:25.107544 (+ 868us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:25.107670 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:25.107714 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:25.107923 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:25.108468 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 11:32:25.108698 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":392,"threads_started":1}
I20260929 11:32:25.110706 13655 log.cc:824] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:25.114435 13655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0: No bootstrap required, opened a new log
I20260929 11:32:25.115733 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:25.125066 13655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a5ed2d733c415194a763ee1286e5d0" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42309 } }
I20260929 11:32:25.125483 13655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:25.125622 13655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a5ed2d733c415194a763ee1286e5d0, State: Initialized, Role: FOLLOWER
I20260929 11:32:25.126411 13655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a5ed2d733c415194a763ee1286e5d0" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42309 } }
I20260929 11:32:25.126719 13655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:25.126873 13655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:25.127068 13655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:25.129168 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:25.121337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49567 (local address 127.23.153.62:42309)
0929 11:32:25.121565 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:25.121572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:25.121918 (+ 346us) server_negotiation.cc:407] Connection header received
0929 11:32:25.122068 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:25.122074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:25.122156 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:25.122273 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:25.122829 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:25.124052 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:25.125541 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:25.126214 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:25.126358 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:25.128150 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:25.128156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:25.128159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:25.128182 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:25.128361 (+ 179us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:25.128404 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:25.128577 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:25.128799 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 11:32:25.128868 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 11:32:25.130648 13655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a5ed2d733c415194a763ee1286e5d0" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42309 } }
I20260929 11:32:25.131362 13655 leader_election.cc:302] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91a5ed2d733c415194a763ee1286e5d0; no voters:
I20260929 11:32:25.132109 13655 leader_election.cc:288] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:25.132413 13665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:25.133574 13665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0 [term 1 LEADER]: Becoming Leader. State: Replica: 91a5ed2d733c415194a763ee1286e5d0, State: Running, Role: LEADER
I20260929 11:32:25.134267 13665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a5ed2d733c415194a763ee1286e5d0" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42309 } }
I20260929 11:32:25.134512 13655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:25.137029 13667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91a5ed2d733c415194a763ee1286e5d0. Latest consensus state: current_term: 1 leader_uuid: "91a5ed2d733c415194a763ee1286e5d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a5ed2d733c415194a763ee1286e5d0" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42309 } } }
I20260929 11:32:25.137671 13667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:25.138576 13666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91a5ed2d733c415194a763ee1286e5d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a5ed2d733c415194a763ee1286e5d0" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42309 } } }
I20260929 11:32:25.138972 13666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:25.139098 13672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:25.145597 13672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:25.157989 13672 catalog_manager.cc:1347] Generated new cluster ID: a639d66dfd684243a96606ff11f66cf3
I20260929 11:32:25.158326 13672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:25.171362 13672 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:25.172962 13672 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:25.184729 13672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91a5ed2d733c415194a763ee1286e5d0: Generated new TSK 0
I20260929 11:32:25.185739 13672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:32:25.199283 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 13588
W20260929 11:32:25.201246 13650 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54329
W20260929 11:32:25.202049 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 13588: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:32:25.202208 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 13588
W20260929 11:32:25.217324 13573 connection.cc:535] client connection to 127.23.153.62:42309 recv error: Network error: failed to read from TLS socket (remote: 127.23.153.62:42309): Connection reset by peer (error 104)
2026-09-29T11:32:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1908 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:25 dist-test-slave-qk05 krb5kdc[13687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:25 dist-test-slave-qk05 krb5kdc[13687](info): set up 2 sockets
Sep 29 11:32:25 dist-test-slave-qk05 krb5kdc[13687](info): commencing operation
krb5kdc: starting...
W20260929 11:32:26.259442 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:26 dist-test-slave-qk05 krb5kdc[13687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:32:26.538244 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:32825
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:39327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:32825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:26.907876 13700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:26.908214 13700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:26.908314 13700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:26.916167 13700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:26.916316 13700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:26.916363 13700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:26.916424 13700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:26.916458 13700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:26.930035 13700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:39327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:32825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:32825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:26.931437 13700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:26.932790 13700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:26.945389 13706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:26.945475 13705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:26.945408 13708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:26.946674 13700 server_base.cc:1034] running on GCE node
I20260929 11:32:26.947695 13700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:26.949318 13700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:26.950598 13700 hybrid_clock.cc:648] HybridClock initialized: now 1790681546950546 us; error 48 us; skew 500 ppm
I20260929 11:32:26.951000 13700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:26 dist-test-slave-qk05 krb5kdc[13687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681546, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:26.954550 13700 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:32:26.956390 13700 webserver.cc:460] Webserver started at http://127.23.153.62:43351/ using document root <none> and password file <none>
I20260929 11:32:26.957165 13700 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:26.957296 13700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:26.957579 13700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:26.961130 13700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "f07b8f5ec2ac4263b5a890ba95d97366"
format_stamp: "Formatted at 2026-09-29 11:32:26 on dist-test-slave-qk05"
server_key: "4db9dec30b8ade160d9d046a901f72ef"
server_key_iv: "6f0f3593d7533f01e4eb90c977731cd8"
server_key_version: "encryptionkey@0"
I20260929 11:32:26.962009 13700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "f07b8f5ec2ac4263b5a890ba95d97366"
format_stamp: "Formatted at 2026-09-29 11:32:26 on dist-test-slave-qk05"
server_key: "4db9dec30b8ade160d9d046a901f72ef"
server_key_iv: "6f0f3593d7533f01e4eb90c977731cd8"
server_key_version: "encryptionkey@0"
I20260929 11:32:26.968305 13700 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 11:32:26.972532 13715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:26.974133 13700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260929 11:32:26.974336 13700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "f07b8f5ec2ac4263b5a890ba95d97366"
format_stamp: "Formatted at 2026-09-29 11:32:26 on dist-test-slave-qk05"
server_key: "4db9dec30b8ade160d9d046a901f72ef"
server_key_iv: "6f0f3593d7533f01e4eb90c977731cd8"
server_key_version: "encryptionkey@0"
I20260929 11:32:26.974540 13700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:26.980912 13700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:26.985519 13700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:26.985886 13700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:27.005017 13700 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:32825
I20260929 11:32:27.005069 13766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:32825 every 8 connection(s)
I20260929 11:32:27.006503 13700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:27.010708 13767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:27.015733 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 13700
I20260929 11:32:27.015906 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:27.016327 24164 external_mini_cluster.cc:1428] Setting key 6793f4e921a0f43c27b72e40ba3558c5
I20260929 11:32:27.022827 13767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366: Bootstrap starting.
I20260929 11:32:27.025331 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:27.018486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46795 (local address 127.23.153.62:32825)
0929 11:32:27.019522 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:27.019537 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:27.019588 (+ 51us) server_negotiation.cc:407] Connection header received
0929 11:32:27.021503 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:27.021531 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:27.021963 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:27.022361 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:27.023598 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:27.023605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:27.023620 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:27.023637 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:27.023706 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:27.023729 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:27.023915 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:27.024030 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 11:32:27.024224 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":426,"threads_started":1}
I20260929 11:32:27.027096 13767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:27.028301 13767 log.cc:824] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:27.029971 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:27.031642 13767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366: No bootstrap required, opened a new log
I20260929 11:32:27.037221 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:27.034437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60877 (local address 127.23.153.62:32825)
0929 11:32:27.034724 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:27.034732 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:27.035292 (+ 560us) server_negotiation.cc:407] Connection header received
0929 11:32:27.035485 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:27.035491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:27.035554 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:27.035635 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:27.036551 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:27.036557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:27.036561 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:27.036583 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:27.036630 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:27.036648 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:27.036765 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:27.036859 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 11:32:27.036934 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 11:32:27.040093 13767 raft_consensus.cc:348] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07b8f5ec2ac4263b5a890ba95d97366" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 32825 } }
I20260929 11:32:27.040531 13767 raft_consensus.cc:374] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:27.040633 13767 raft_consensus.cc:729] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07b8f5ec2ac4263b5a890ba95d97366, State: Initialized, Role: FOLLOWER
I20260929 11:32:27.041252 13767 consensus_queue.cc:260] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07b8f5ec2ac4263b5a890ba95d97366" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 32825 } }
I20260929 11:32:27.041507 13767 raft_consensus.cc:388] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:27.041644 13767 raft_consensus.cc:482] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:27.041805 13767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:27.044553 13767 raft_consensus.cc:504] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07b8f5ec2ac4263b5a890ba95d97366" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 32825 } }
I20260929 11:32:27.045264 13767 leader_election.cc:302] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f07b8f5ec2ac4263b5a890ba95d97366; no voters:
I20260929 11:32:27.045922 13767 leader_election.cc:288] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:27.046034 13777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:27.046725 13777 raft_consensus.cc:686] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366 [term 1 LEADER]: Becoming Leader. State: Replica: f07b8f5ec2ac4263b5a890ba95d97366, State: Running, Role: LEADER
I20260929 11:32:27.047559 13777 consensus_queue.cc:237] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07b8f5ec2ac4263b5a890ba95d97366" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 32825 } }
I20260929 11:32:27.048456 13767 sys_catalog.cc:564] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:27.051048 13778 sys_catalog.cc:455] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f07b8f5ec2ac4263b5a890ba95d97366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07b8f5ec2ac4263b5a890ba95d97366" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 32825 } } }
I20260929 11:32:27.051591 13778 sys_catalog.cc:458] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:27.050706 13779 sys_catalog.cc:455] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f07b8f5ec2ac4263b5a890ba95d97366. Latest consensus state: current_term: 1 leader_uuid: "f07b8f5ec2ac4263b5a890ba95d97366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07b8f5ec2ac4263b5a890ba95d97366" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 32825 } } }
I20260929 11:32:27.051918 13779 sys_catalog.cc:458] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:27.052655 13786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:27.057853 13786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:27.069144 13786 catalog_manager.cc:1347] Generated new cluster ID: cda6f1bf1e654870823c2cd8f795989e
I20260929 11:32:27.069363 13786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:27.089803 13786 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:27.090322 13786 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:27.105647 13786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f07b8f5ec2ac4263b5a890ba95d97366: Generated new TSK 0
I20260929 11:32:27.107081 13786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:32:27.117154 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 13700
W20260929 11:32:27.118582 13762 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46795
W20260929 11:32:27.119524 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 13700: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:32:27.119680 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 13700
2026-09-29T11:32:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1919 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:27 dist-test-slave-qk05 krb5kdc[13798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:27 dist-test-slave-qk05 krb5kdc[13798](info): set up 2 sockets
Sep 29 11:32:27 dist-test-slave-qk05 krb5kdc[13798](info): commencing operation
krb5kdc: starting...
W20260929 11:32:28.179996 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:28 dist-test-slave-qk05 krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:32:28.467993 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:45191
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:45191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:28.841102 13811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:28.841432 13811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:28.841590 13811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:28.849406 13811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:28.849550 13811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:28.849611 13811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:28.849700 13811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:28.849758 13811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:28.861693 13811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:45191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:45191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:28.863222 13811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:28.864676 13811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:28.878625 13816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:28.878680 13817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:28.878674 13819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:28.879843 13811 server_base.cc:1034] running on GCE node
I20260929 11:32:28.880743 13811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:28.882467 13811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:28.883704 13811 hybrid_clock.cc:648] HybridClock initialized: now 1790681548883645 us; error 57 us; skew 500 ppm
I20260929 11:32:28.884097 13811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:28 dist-test-slave-qk05 krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681548, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:28.887418 13811 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:32:28.889106 13811 webserver.cc:460] Webserver started at http://127.23.153.62:33627/ using document root <none> and password file <none>
I20260929 11:32:28.889866 13811 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:28.890009 13811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:28.890331 13811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:28.893888 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "5a432818f8bf46e58ef646b4aaac5538"
format_stamp: "Formatted at 2026-09-29 11:32:28 on dist-test-slave-qk05"
server_key: "ef913de6d11cb0aea361edc1937f2e09"
server_key_iv: "9c57481ad36f2ff398ae3134a8eeb1fe"
server_key_version: "encryptionkey@0"
I20260929 11:32:28.894824 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "5a432818f8bf46e58ef646b4aaac5538"
format_stamp: "Formatted at 2026-09-29 11:32:28 on dist-test-slave-qk05"
server_key: "ef913de6d11cb0aea361edc1937f2e09"
server_key_iv: "9c57481ad36f2ff398ae3134a8eeb1fe"
server_key_version: "encryptionkey@0"
I20260929 11:32:28.900849 13811 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 11:32:28.905182 13826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:28.906872 13811 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 11:32:28.907125 13811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "5a432818f8bf46e58ef646b4aaac5538"
format_stamp: "Formatted at 2026-09-29 11:32:28 on dist-test-slave-qk05"
server_key: "ef913de6d11cb0aea361edc1937f2e09"
server_key_iv: "9c57481ad36f2ff398ae3134a8eeb1fe"
server_key_version: "encryptionkey@0"
I20260929 11:32:28.907348 13811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:28.930095 13811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:28.934981 13811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:28.935349 13811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:28.955633 13877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:45191 every 8 connection(s)
I20260929 11:32:28.955628 13811 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:45191
I20260929 11:32:28.957304 13811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:28.961757 13878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:28.966995 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 13811
I20260929 11:32:28.967221 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:28.967756 24164 external_mini_cluster.cc:1428] Setting key c5bb17ccfb369a84894bc7ebb9550423
I20260929 11:32:28.975322 13878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538: Bootstrap starting.
I20260929 11:32:28.978972 13878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:28.980208 13878 log.cc:824] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:28.984614 13878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538: No bootstrap required, opened a new log
Sep 29 11:32:28 dist-test-slave-qk05 krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:32:28.996524 13878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a432818f8bf46e58ef646b4aaac5538" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45191 } }
I20260929 11:32:28.997099 13878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:28.997341 13878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a432818f8bf46e58ef646b4aaac5538, State: Initialized, Role: FOLLOWER
I20260929 11:32:28.998062 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:28.970040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38505 (local address 127.23.153.62:45191)
0929 11:32:28.970881 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:28.970895 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:28.970956 (+ 61us) server_negotiation.cc:407] Connection header received
0929 11:32:28.973707 (+ 2751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:28.973745 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:28.974288 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:28.974757 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:28.977125 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:28.978816 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:28.980729 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:28.981981 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:28.982217 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:28.987002 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:28.987046 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:28.987073 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:28.987138 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:28.991742 (+ 4604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:28.992866 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:28.992876 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:28.992886 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:28.993173 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:28.993725 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:28.993730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:28.993735 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:28.994828 (+ 1093us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:28.995339 (+ 511us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:28.995714 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 11:32:28.996340 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":349,"threads_started":1}
I20260929 11:32:28.998214 13878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a432818f8bf46e58ef646b4aaac5538" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45191 } }
I20260929 11:32:28.998680 13878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:28.998859 13878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:28.999132 13878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:29.002463 13878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a432818f8bf46e58ef646b4aaac5538" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45191 } }
I20260929 11:32:29.002745 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:29.003167 13878 leader_election.cc:302] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a432818f8bf46e58ef646b4aaac5538; no voters:
I20260929 11:32:29.003808 13878 leader_election.cc:288] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:29.003953 13883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:29.005990 13878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:29.009064 13883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538 [term 1 LEADER]: Becoming Leader. State: Replica: 5a432818f8bf46e58ef646b4aaac5538, State: Running, Role: LEADER
I20260929 11:32:29.010077 13883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a432818f8bf46e58ef646b4aaac5538" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45191 } }
I20260929 11:32:29.013525 13885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a432818f8bf46e58ef646b4aaac5538. Latest consensus state: current_term: 1 leader_uuid: "5a432818f8bf46e58ef646b4aaac5538" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a432818f8bf46e58ef646b4aaac5538" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45191 } } }
I20260929 11:32:29.013995 13885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:29.014662 13884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a432818f8bf46e58ef646b4aaac5538" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a432818f8bf46e58ef646b4aaac5538" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 45191 } } }
I20260929 11:32:29.014994 13884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:29.020332 13894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:29.027323 13894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:29.030656 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:29.009674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60825 (local address 127.23.153.62:45191)
0929 11:32:29.009843 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:29.009850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:29.010337 (+ 487us) server_negotiation.cc:407] Connection header received
0929 11:32:29.010528 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:29.010535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:29.010640 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:29.010765 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:29.013211 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:29.014351 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:29.016353 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:29.017080 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:29.017352 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:29.021187 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:29.021198 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:29.021204 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:29.021263 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:29.025653 (+ 4390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:29.026829 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:29.026836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:29.026839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:29.027069 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:29.027563 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:29.027569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:29.027572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:29.028113 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:29.028418 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:29.029950 (+ 1532us) server_negotiation.cc:299] Negotiation successful
0929 11:32:29.030225 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 11:32:29.042296 13894 catalog_manager.cc:1347] Generated new cluster ID: 2cd6a95216834df785594ef9d9cf04ff
I20260929 11:32:29.042430 13894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:29.076076 13894 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:29.078547 13894 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:29.093032 13894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a432818f8bf46e58ef646b4aaac5538: Generated new TSK 0
I20260929 11:32:29.093993 13894 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:32:29.108177 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 13811
I20260929 11:32:29.110126 13873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:29.212368 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 13811
2026-09-29T11:32:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2091 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:29 dist-test-slave-qk05 krb5kdc[13910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:29 dist-test-slave-qk05 krb5kdc[13910](info): set up 2 sockets
Sep 29 11:32:29 dist-test-slave-qk05 krb5kdc[13910](info): commencing operation
krb5kdc: starting...
W20260929 11:32:30.271530 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:30 dist-test-slave-qk05 krb5kdc[13910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:32:30.556725 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:42749
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:35875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:42749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:30.945554 13923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:30.945979 13923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:30.946244 13923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:30.954900 13923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:30.955083 13923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:30.955148 13923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:30.955219 13923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:30.955268 13923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:30.967237 13923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:35875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:42749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:42749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:30.968623 13923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:30.970026 13923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:30.983052 13931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:30.983279 13929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:30.983294 13928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:30.984683 13923 server_base.cc:1034] running on GCE node
I20260929 11:32:30.985868 13923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:30.987645 13923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:30.988925 13923 hybrid_clock.cc:648] HybridClock initialized: now 1790681550988789 us; error 128 us; skew 500 ppm
I20260929 11:32:30.989434 13923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:30 dist-test-slave-qk05 krb5kdc[13910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681550, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:30.997370 13923 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:32:30.999497 13923 webserver.cc:460] Webserver started at http://127.23.153.62:34853/ using document root <none> and password file <none>
I20260929 11:32:31.000448 13923 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:31.000604 13923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:31.000958 13923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:31.004535 13923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "b623e1bc471844ca8f4c1b59cdb10ec7"
format_stamp: "Formatted at 2026-09-29 11:32:30 on dist-test-slave-qk05"
server_key: "16c2ad30db70e50aaf0a6fa26112e7d6"
server_key_iv: "cb7adf25e2cd016708da6af9bc008a17"
server_key_version: "encryptionkey@0"
I20260929 11:32:31.005442 13923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "b623e1bc471844ca8f4c1b59cdb10ec7"
format_stamp: "Formatted at 2026-09-29 11:32:30 on dist-test-slave-qk05"
server_key: "16c2ad30db70e50aaf0a6fa26112e7d6"
server_key_iv: "cb7adf25e2cd016708da6af9bc008a17"
server_key_version: "encryptionkey@0"
I20260929 11:32:31.011787 13923 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 11:32:31.017827 13938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:31.019721 13923 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:32:31.019956 13923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "b623e1bc471844ca8f4c1b59cdb10ec7"
format_stamp: "Formatted at 2026-09-29 11:32:30 on dist-test-slave-qk05"
server_key: "16c2ad30db70e50aaf0a6fa26112e7d6"
server_key_iv: "cb7adf25e2cd016708da6af9bc008a17"
server_key_version: "encryptionkey@0"
I20260929 11:32:31.020255 13923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:31.036585 13923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:31.042147 13923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:31.042449 13923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:31.063519 13923 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:42749
I20260929 11:32:31.063547 13989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:42749 every 8 connection(s)
I20260929 11:32:31.065040 13923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:31.065857 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 13923
I20260929 11:32:31.066032 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:31.066586 24164 external_mini_cluster.cc:1428] Setting key 3ce8871af15acf20852045884b38cdfc
I20260929 11:32:31.070130 13990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:31.081670 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:31.068742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55371 (local address 127.23.153.62:42749)
0929 11:32:31.069626 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:31.069642 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:31.069686 (+ 44us) server_negotiation.cc:407] Connection header received
0929 11:32:31.071507 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:31.071534 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:31.071990 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:31.072412 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:31.073025 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:31.074458 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:31.075906 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:31.076677 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:31.076879 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:31.078173 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:31.078183 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:31.078201 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:31.078852 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:31.078948 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:31.078987 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:31.079239 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:31.079747 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 11:32:31.080217 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":405,"threads_started":1}
I20260929 11:32:31.082626 13990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7: Bootstrap starting.
I20260929 11:32:31.087046 13990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:31.087632 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:31.088565 13990 log.cc:824] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:31.092160 13990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7: No bootstrap required, opened a new log
I20260929 11:32:31.098548 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:31.092304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57523 (local address 127.23.153.62:42749)
0929 11:32:31.092487 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:31.092493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:31.092903 (+ 410us) server_negotiation.cc:407] Connection header received
0929 11:32:31.093101 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:31.093106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:31.093181 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:31.093270 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:31.093773 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:31.094549 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:31.095994 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:31.096525 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:31.096692 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:31.097770 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:31.097776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:31.097782 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:31.097808 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:31.097851 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:31.097867 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:31.097961 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:31.098094 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 11:32:31.098200 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 11:32:31.100780 13990 raft_consensus.cc:348] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b623e1bc471844ca8f4c1b59cdb10ec7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42749 } }
I20260929 11:32:31.101236 13990 raft_consensus.cc:374] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:31.101311 13990 raft_consensus.cc:729] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b623e1bc471844ca8f4c1b59cdb10ec7, State: Initialized, Role: FOLLOWER
I20260929 11:32:31.102015 13990 consensus_queue.cc:260] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b623e1bc471844ca8f4c1b59cdb10ec7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42749 } }
I20260929 11:32:31.102309 13990 raft_consensus.cc:388] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:31.102455 13990 raft_consensus.cc:482] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:31.102648 13990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:31.105407 13990 raft_consensus.cc:504] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b623e1bc471844ca8f4c1b59cdb10ec7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42749 } }
I20260929 11:32:31.106040 13990 leader_election.cc:302] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b623e1bc471844ca8f4c1b59cdb10ec7; no voters:
I20260929 11:32:31.106932 13990 leader_election.cc:288] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:31.107249 14000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:31.108660 14000 raft_consensus.cc:686] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7 [term 1 LEADER]: Becoming Leader. State: Replica: b623e1bc471844ca8f4c1b59cdb10ec7, State: Running, Role: LEADER
I20260929 11:32:31.109377 14000 consensus_queue.cc:237] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b623e1bc471844ca8f4c1b59cdb10ec7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42749 } }
I20260929 11:32:31.109948 13990 sys_catalog.cc:564] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:31.113461 14001 sys_catalog.cc:455] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b623e1bc471844ca8f4c1b59cdb10ec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b623e1bc471844ca8f4c1b59cdb10ec7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42749 } } }
I20260929 11:32:31.113915 14001 sys_catalog.cc:458] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:31.116194 14002 sys_catalog.cc:455] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b623e1bc471844ca8f4c1b59cdb10ec7. Latest consensus state: current_term: 1 leader_uuid: "b623e1bc471844ca8f4c1b59cdb10ec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b623e1bc471844ca8f4c1b59cdb10ec7" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 42749 } } }
I20260929 11:32:31.116595 14002 sys_catalog.cc:458] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:31.117954 14007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:31.125823 14007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:31.135999 14007 catalog_manager.cc:1347] Generated new cluster ID: 660187fa2fbb4a15a375e3546380b8f3
I20260929 11:32:31.136173 14007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:31.170889 14007 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:31.172408 14007 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:31.191623 14007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b623e1bc471844ca8f4c1b59cdb10ec7: Generated new TSK 0
I20260929 11:32:31.193117 14007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:32:31.212702 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 13923
W20260929 11:32:31.214913 13985 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55371
W20260929 11:32:31.215806 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 13923: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:32:31.215972 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 13923
2026-09-29T11:32:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2005 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:31 dist-test-slave-qk05 krb5kdc[14021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:31 dist-test-slave-qk05 krb5kdc[14021](info): set up 2 sockets
Sep 29 11:32:31 dist-test-slave-qk05 krb5kdc[14021](info): commencing operation
krb5kdc: starting...
W20260929 11:32:32.267540 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:32 dist-test-slave-qk05 krb5kdc[14021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:32:32.551196 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:39225
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:38051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:39225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:32.908334 14034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:32.908653 14034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:32.908772 14034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:32.916936 14034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:32.917083 14034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:32.917129 14034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:32.917193 14034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:32.917258 14034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:32.928579 14034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:38051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:39225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:39225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:32.930560 14034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:32.932114 14034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:32.944882 14039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:32.944811 14042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:32.944854 14040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:32.945724 14034 server_base.cc:1034] running on GCE node
I20260929 11:32:32.946584 14034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:32.948268 14034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:32.949539 14034 hybrid_clock.cc:648] HybridClock initialized: now 1790681552949473 us; error 59 us; skew 500 ppm
I20260929 11:32:32.949968 14034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:32 dist-test-slave-qk05 krb5kdc[14021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681552, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:32.954512 14034 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:32:32.956290 14034 webserver.cc:460] Webserver started at http://127.23.153.62:43175/ using document root <none> and password file <none>
I20260929 11:32:32.957006 14034 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:32.957141 14034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:32.957446 14034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:32.960860 14034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "e4863fcb370743dd8145293fcf60f5e4"
format_stamp: "Formatted at 2026-09-29 11:32:32 on dist-test-slave-qk05"
server_key: "e320e3cbef633459c6b32ab9b95d1066"
server_key_iv: "47df78378117227be7deae60beb74d4d"
server_key_version: "encryptionkey@0"
I20260929 11:32:32.961714 14034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "e4863fcb370743dd8145293fcf60f5e4"
format_stamp: "Formatted at 2026-09-29 11:32:32 on dist-test-slave-qk05"
server_key: "e320e3cbef633459c6b32ab9b95d1066"
server_key_iv: "47df78378117227be7deae60beb74d4d"
server_key_version: "encryptionkey@0"
I20260929 11:32:32.967944 14034 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 11:32:32.973651 14049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:32.975561 14034 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:32:32.975801 14034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "e4863fcb370743dd8145293fcf60f5e4"
format_stamp: "Formatted at 2026-09-29 11:32:32 on dist-test-slave-qk05"
server_key: "e320e3cbef633459c6b32ab9b95d1066"
server_key_iv: "47df78378117227be7deae60beb74d4d"
server_key_version: "encryptionkey@0"
I20260929 11:32:32.976033 14034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:32.983814 14034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:32.988818 14034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:32.989158 14034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:33.007829 14034 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:39225
I20260929 11:32:33.007859 14100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:39225 every 8 connection(s)
I20260929 11:32:33.009395 14034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:33.013710 14101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:33.019125 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 14034
I20260929 11:32:33.019351 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:33.019943 24164 external_mini_cluster.cc:1428] Setting key c90ac9e1c5491e73ec99009393773a4c
I20260929 11:32:33.027283 14101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4: Bootstrap starting.
I20260929 11:32:33.029995 14104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:33.022183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53607 (local address 127.23.153.62:39225)
0929 11:32:33.023151 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:33.023167 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:33.023217 (+ 50us) server_negotiation.cc:407] Connection header received
0929 11:32:33.025521 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:33.025554 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:33.026071 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:33.026580 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:33.027719 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:33.027728 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:33.027745 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:33.027770 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:33.027845 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:33.027871 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:33.028096 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:33.028571 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 11:32:33.028821 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":422,"threads_started":1}
I20260929 11:32:33.031510 14101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:33.032760 14101 log.cc:824] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:33.034881 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:33.036988 14101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4: No bootstrap required, opened a new log
I20260929 11:32:33.042069 14104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:33.039598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44503 (local address 127.23.153.62:39225)
0929 11:32:33.039807 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:33.039814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:33.040086 (+ 272us) server_negotiation.cc:407] Connection header received
0929 11:32:33.040242 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:33.040248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:33.040330 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:33.040462 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:33.041411 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:33.041418 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:33.041423 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 11:32:33.041443 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:33.041504 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 11:32:33.041528 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:33.041662 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:33.041707 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 11:32:33.041766 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 11:32:33.046273 14101 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4863fcb370743dd8145293fcf60f5e4" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39225 } }
I20260929 11:32:33.046706 14101 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 11:32:33.046774 14101 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4863fcb370743dd8145293fcf60f5e4, State: Initialized, Role: FOLLOWER
I20260929 11:32:33.047394 14101 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4863fcb370743dd8145293fcf60f5e4" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39225 } }
I20260929 11:32:33.047631 14101 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 11:32:33.047734 14101 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 11:32:33.047866 14101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:33.050168 14101 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4863fcb370743dd8145293fcf60f5e4" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39225 } }
I20260929 11:32:33.050745 14101 leader_election.cc:302] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4863fcb370743dd8145293fcf60f5e4; no voters:
I20260929 11:32:33.051378 14101 leader_election.cc:288] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 11:32:33.051523 14111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:33.052234 14111 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4 [term 1 LEADER]: Becoming Leader. State: Replica: e4863fcb370743dd8145293fcf60f5e4, State: Running, Role: LEADER
I20260929 11:32:33.052841 14111 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4863fcb370743dd8145293fcf60f5e4" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39225 } }
I20260929 11:32:33.053572 14101 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:33.055950 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4863fcb370743dd8145293fcf60f5e4. Latest consensus state: current_term: 1 leader_uuid: "e4863fcb370743dd8145293fcf60f5e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4863fcb370743dd8145293fcf60f5e4" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39225 } } }
I20260929 11:32:33.055932 14112 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4863fcb370743dd8145293fcf60f5e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4863fcb370743dd8145293fcf60f5e4" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39225 } } }
I20260929 11:32:33.056525 14112 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:33.056525 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:33.059696 14117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:33.064842 14117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:33.074661 14117 catalog_manager.cc:1347] Generated new cluster ID: 22f6f97ecdbf48e292acfa177167e935
I20260929 11:32:33.074790 14117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:33.086228 14117 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:33.086717 14117 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:33.097846 14117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4863fcb370743dd8145293fcf60f5e4: Generated new TSK 0
I20260929 11:32:33.098809 14117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 11:32:33.108561 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 14034
W20260929 11:32:33.109977 14096 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53607
W20260929 11:32:33.110912 24164 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 14034: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 11:32:33.111115 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 14034
2026-09-29T11:32:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1891 ms)
[----------] 18 tests from AuthTokenIssuingTest (35614 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:33 dist-test-slave-qk05 krb5kdc[14132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:33 dist-test-slave-qk05 krb5kdc[14132](info): set up 2 sockets
Sep 29 11:32:33 dist-test-slave-qk05 krb5kdc[14132](info): commencing operation
krb5kdc: starting...
W20260929 11:32:34.168303 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:34 dist-test-slave-qk05 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:32:34.442541 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:40527
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:40527,127.23.153.61:35839,127.23.153.60:35425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:34.804575 14145 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 11:32:34.804950 14145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:34.805027 14145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:34.805119 14145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:34.812865 14145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:34.812983 14145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:34.813040 14145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:34.813089 14145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:34.813151 14145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:34.823937 14145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:40527,127.23.153.61:35839,127.23.153.60:35425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:40527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:34.825361 14145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:34.826754 14145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:34.840469 14151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:34.840431 14153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:34.840499 14150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:34.841509 14145 server_base.cc:1034] running on GCE node
I20260929 11:32:34.842360 14145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:34.843987 14145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:34.845269 14145 hybrid_clock.cc:648] HybridClock initialized: now 1790681554845197 us; error 40 us; skew 500 ppm
I20260929 11:32:34.845626 14145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:34 dist-test-slave-qk05 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681554, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:34.848994 14145 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:32:34.850765 14145 webserver.cc:460] Webserver started at http://127.23.153.62:44479/ using document root <none> and password file <none>
I20260929 11:32:34.851614 14145 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:34.851728 14145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:34.852022 14145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:34.855455 14145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "b83f991d5436402697bca98b46050569"
format_stamp: "Formatted at 2026-09-29 11:32:34 on dist-test-slave-qk05"
server_key: "d41bc12ca8d3eff0026f4d4ea02ee909"
server_key_iv: "e54dc063840d44ad64afa221694757d5"
server_key_version: "encryptionkey@0"
I20260929 11:32:34.856392 14145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "b83f991d5436402697bca98b46050569"
format_stamp: "Formatted at 2026-09-29 11:32:34 on dist-test-slave-qk05"
server_key: "d41bc12ca8d3eff0026f4d4ea02ee909"
server_key_iv: "e54dc063840d44ad64afa221694757d5"
server_key_version: "encryptionkey@0"
I20260929 11:32:34.862740 14145 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 11:32:34.867635 14160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:34.869568 14145 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260929 11:32:34.869885 14145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "b83f991d5436402697bca98b46050569"
format_stamp: "Formatted at 2026-09-29 11:32:34 on dist-test-slave-qk05"
server_key: "d41bc12ca8d3eff0026f4d4ea02ee909"
server_key_iv: "e54dc063840d44ad64afa221694757d5"
server_key_version: "encryptionkey@0"
I20260929 11:32:34.870229 14145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:34.907147 14145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:34.912813 14145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:34.913097 14145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:34.934227 14145 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:40527
I20260929 11:32:34.934317 14211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:40527 every 8 connection(s)
I20260929 11:32:34.936602 14145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:34.941155 14212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:34.942080 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 14145
I20260929 11:32:34.942358 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:34.943058 24164 external_mini_cluster.cc:1428] Setting key fe31eb0682f9c5da284567648a04c323
I20260929 11:32:34.949770 14212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.153.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.61@KRBTEST.COM" created.
Sep 29 11:32:34 dist-test-slave-qk05 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681554, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:32:34.982003 14215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:34.954847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.23.153.62:40527)
0929 11:32:34.955957 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:34.955979 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:34.956580 (+ 601us) server_negotiation.cc:407] Connection header received
0929 11:32:34.959741 (+ 3161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:34.959750 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:34.960178 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:34.960392 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:34.962221 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:34.963567 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:34.966270 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:34.967105 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:34.967525 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:34.972925 (+ 5400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:34.972934 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:34.972949 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:34.972992 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:34.977153 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:34.977957 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:34.977964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:34.977966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:34.978133 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:34.978551 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:34.978555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:34.978557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:34.979173 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:34.979371 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:34.979954 (+ 583us) server_negotiation.cc:299] Negotiation successful
0929 11:32:34.980395 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":792,"thread_start_us":425,"threads_started":1}
I20260929 11:32:34.982023 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:34.954912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:40527 (local address 127.0.0.1:55672)
0929 11:32:34.956362 (+ 1450us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:34.956411 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:34.959168 (+ 2757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:34.960474 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:34.960482 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:34.961355 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:34.961858 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:34.961886 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:34.963796 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:34.963803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:34.965858 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:34.965887 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:34.967599 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:34.967609 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:34.967888 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:34.968949 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:34.968980 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:34.972619 (+ 3639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:34.977335 (+ 4716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:34.977349 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:34.977364 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:34.977807 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:34.978263 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:34.978267 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:34.978271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:34.978439 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:34.979329 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:34.979337 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:34.979560 (+ 223us) client_negotiation.cc:769] Sending connection context
0929 11:32:34.979807 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 11:32:34.980544 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"mutex_wait_us":235,"spinlock_wait_cycles":17792,"thread_start_us":871,"threads_started":1}
I20260929 11:32:34.985875 14212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:32:34.987241 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:34.986646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35839 (local address 127.0.0.1:40662)
0929 11:32:34.986854 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:34.987058 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 11:32:34.987592 14161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35839: connect: Connection refused (error 111)
I20260929 11:32:34.989423 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:34.988905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35839 (local address 127.0.0.1:40668)
0929 11:32:34.989130 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:34.989234 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 11:32:34.989794 14212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.61:35839: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35839: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.61.keytab.
Entry for principal kudu/127.23.153.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.61.keytab.
I20260929 11:32:35.007411 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.61:35839
--webserver_interface=127.23.153.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.61.keytab
--principal=kudu/127.23.153.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:40527,127.23.153.61:35839,127.23.153.60:35425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:32:35.039484 14212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } attempt: 1
I20260929 11:32:35.040817 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.040258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35839 (local address 127.0.0.1:40674)
0929 11:32:35.040496 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.040651 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 11:32:35.041877 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.041461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35839 (local address 127.0.0.1:40678)
0929 11:32:35.041617 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.041709 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 11:32:35.042095 14212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.61:35839: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35839: connect: Connection refused (error 111)
I20260929 11:32:35.110443 14212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } attempt: 2
I20260929 11:32:35.111807 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.111229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35839 (local address 127.0.0.1:40690)
0929 11:32:35.111503 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.111628 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 11:32:35.113031 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.112555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35839 (local address 127.0.0.1:40698)
0929 11:32:35.112748 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.112847 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 11:32:35.113281 14212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.61:35839: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35839: connect: Connection refused (error 111)
I20260929 11:32:35.204591 14212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } attempt: 3
I20260929 11:32:35.205916 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.205360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35839 (local address 127.0.0.1:40702)
0929 11:32:35.205613 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.205746 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 11:32:35.207113 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.206612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35839 (local address 127.0.0.1:40706)
0929 11:32:35.206782 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.206878 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 11:32:35.207357 14212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.61:35839: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35839: connect: Connection refused (error 111)
I20260929 11:32:35.350701 14212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } attempt: 4
I20260929 11:32:35.352574 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.351705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35839 (local address 127.0.0.1:40718)
0929 11:32:35.352190 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.352339 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256}
I20260929 11:32:35.354156 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.353521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35839 (local address 127.0.0.1:40732)
0929 11:32:35.353778 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.353918 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260929 11:32:35.354578 14212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.61:35839: Network error: Client connection negotiation failed: client connection to 127.23.153.61:35839: connect: Connection refused (error 111)
W20260929 11:32:35.383550 14219 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 11:32:35.383972 14219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:35.384078 14219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:35.384336 14219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:35.392747 14219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:35.392887 14219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:35.392973 14219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:35.393054 14219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:35.393144 14219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:35.404363 14219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:40527,127.23.153.61:35839,127.23.153.60:35425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.61.keytab
--principal=kudu/127.23.153.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.61:35839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.153.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:35.405830 14219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:35.407377 14219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:35.422546 14224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:35.422669 14225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:35.423147 14227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:35.424510 14219 server_base.cc:1034] running on GCE node
I20260929 11:32:35.425412 14219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:35.427192 14219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:35.428475 14219 hybrid_clock.cc:648] HybridClock initialized: now 1790681555428406 us; error 38 us; skew 500 ppm
I20260929 11:32:35.428884 14219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:35 dist-test-slave-qk05 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681555, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:35.440066 14219 init.cc:377] Logged in from keytab as kudu/127.23.153.61@KRBTEST.COM (short username kudu)
I20260929 11:32:35.441956 14219 webserver.cc:460] Webserver started at http://127.23.153.61:40909/ using document root <none> and password file <none>
I20260929 11:32:35.442737 14219 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:35.442850 14219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:35.443246 14219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:35.446477 14219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-1/data/instance:
uuid: "ad3beb5c821b459281fb9b769517017d"
format_stamp: "Formatted at 2026-09-29 11:32:35 on dist-test-slave-qk05"
server_key: "2e96a5ddd708d9df9a8c80ca92413ffc"
server_key_iv: "dc8e60743685b31816f0d2168cc2ec45"
server_key_version: "encryptionkey@0"
I20260929 11:32:35.447434 14219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-1/wal/instance:
uuid: "ad3beb5c821b459281fb9b769517017d"
format_stamp: "Formatted at 2026-09-29 11:32:35 on dist-test-slave-qk05"
server_key: "2e96a5ddd708d9df9a8c80ca92413ffc"
server_key_iv: "dc8e60743685b31816f0d2168cc2ec45"
server_key_version: "encryptionkey@0"
I20260929 11:32:35.453349 14219 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:32:35.457356 14234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:35.459108 14219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 11:32:35.459321 14219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-1/wal
uuid: "ad3beb5c821b459281fb9b769517017d"
format_stamp: "Formatted at 2026-09-29 11:32:35 on dist-test-slave-qk05"
server_key: "2e96a5ddd708d9df9a8c80ca92413ffc"
server_key_iv: "dc8e60743685b31816f0d2168cc2ec45"
server_key_version: "encryptionkey@0"
I20260929 11:32:35.459570 14219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:35.480669 14219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:35.485754 14219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:35.486057 14219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:35.507858 14219 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.61:35839
I20260929 11:32:35.507879 14285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.61:35839 every 8 connection(s)
I20260929 11:32:35.509465 14219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-1/data/info.pb
I20260929 11:32:35.513747 14286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:35.519141 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 14219
I20260929 11:32:35.519471 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-1/wal/instance
I20260929 11:32:35.520051 24164 external_mini_cluster.cc:1428] Setting key 04bc8ff7fd22f3f5b0a6aae0b86b15d6
I20260929 11:32:35.522702 14286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:32:35 dist-test-slave-qk05 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681555, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.61@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:32:35.548552 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.526346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.23.153.62:40527)
0929 11:32:35.527053 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:35.527059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:35.528339 (+ 1280us) server_negotiation.cc:407] Connection header received
0929 11:32:35.531133 (+ 2794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:35.531139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:35.531239 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:35.531348 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:35.533403 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:35.534697 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:35.536500 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:35.537108 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:35.537253 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:35.541043 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:35.541049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:35.541054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:35.541114 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:35.544825 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:35.545826 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:35.545833 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:35.545837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:35.546014 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:35.546439 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:35.546444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:35.546448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:35.546876 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:35.547086 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:35.547866 (+ 780us) server_negotiation.cc:299] Negotiation successful
0929 11:32:35.548136 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":334,"threads_started":1}
WARNING: no policy specified for kudu/127.23.153.60@KRBTEST.COM; defaulting to no policy
I20260929 11:32:35.549304 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.527038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:40527 (local address 127.0.0.1:55686)
0929 11:32:35.528028 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.528093 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:35.530679 (+ 2586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:35.531681 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:35.531695 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:35.532361 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:35.533065 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:35.533094 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:35.534909 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:35.534914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:35.536312 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:35.536330 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:35.537378 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:35.537383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:35.537625 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:35.538343 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:35.538369 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:35.540731 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:35.545044 (+ 4313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:35.545054 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:35.545072 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:35.545643 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:35.546126 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:35.546129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:35.546131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:35.546319 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:35.546996 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:35.547002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:35.547445 (+ 443us) client_negotiation.cc:769] Sending connection context
0929 11:32:35.547697 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 11:32:35.548140 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":398,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.60@KRBTEST.COM" created.
I20260929 11:32:35.551442 14286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:32:35 dist-test-slave-qk05 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681555, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.61@KRBTEST.COM for kudu/127.23.153.61@KRBTEST.COM
I20260929 11:32:35.567866 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.552312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35839 (local address 127.0.0.1:40748)
0929 11:32:35.552514 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.552534 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:35.552662 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:35.553808 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:35.553812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:35.554224 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:35.554380 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:35.554389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:35.556339 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:35.556343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:35.557368 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:35.557385 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:35.558574 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:35.558580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:35.558713 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:35.559402 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:35.559421 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:35.561737 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:35.565172 (+ 3435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:35.565178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:35.565182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:35.565649 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:35.566057 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:35.566062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:35.566064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:35.566189 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:35.566820 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:35.566824 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:35.566961 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 11:32:35.567129 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 11:32:35.567444 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 11:32:35.567948 14291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.552549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40748 (local address 127.23.153.61:35839)
0929 11:32:35.553220 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:35.553228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:35.553261 (+ 33us) server_negotiation.cc:407] Connection header received
0929 11:32:35.553319 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:35.553327 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:35.553593 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:35.553759 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:35.554568 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:35.556061 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:35.557543 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:35.558409 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:35.558535 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:35.561963 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:35.561978 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:35.561994 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:35.562033 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:35.565003 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:35.565795 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:35.565802 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:35.565805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:35.565932 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:35.566291 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:35.566295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:35.566297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:35.566673 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:35.566889 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:35.567282 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 11:32:35.567566 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":485,"thread_start_us":355,"threads_started":1}
I20260929 11:32:35.571313 14286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:32:35.572724 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.572205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:60948)
0929 11:32:35.572409 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.572549 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260929 11:32:35.573053 14236 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
I20260929 11:32:35.574760 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.574356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:60960)
0929 11:32:35.574548 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.574633 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 11:32:35.574982 14286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:35425: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.60.keytab.
Entry for principal kudu/127.23.153.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.60.keytab.
I20260929 11:32:35.586563 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.60:35425
--webserver_interface=127.23.153.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.60.keytab
--principal=kudu/127.23.153.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:40527,127.23.153.61:35839,127.23.153.60:35425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:32:35.624356 14286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } attempt: 1
I20260929 11:32:35.626490 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.625456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:60964)
0929 11:32:35.626082 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.626260 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414}
I20260929 11:32:35.628181 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.627690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:60972)
0929 11:32:35.627865 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.627992 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 11:32:35.628472 14286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:35425: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
I20260929 11:32:35.653934 14212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } attempt: 5
Sep 29 11:32:35 dist-test-slave-qk05 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681554, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for kudu/127.23.153.61@KRBTEST.COM
I20260929 11:32:35.670024 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.654802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35839 (local address 127.0.0.1:40752)
0929 11:32:35.655102 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.655123 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:35.655284 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:35.655649 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:35.655654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:35.656246 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:35.656443 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:35.656454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:35.657635 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:35.657648 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:35.658545 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:35.658561 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:35.659445 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:35.659452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:35.659676 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:35.660402 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:35.660424 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:35.662857 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:35.666836 (+ 3979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:35.666842 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:35.666847 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:35.667630 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:35.668283 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:35.668288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:35.668293 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:35.668463 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:35.669017 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:35.669022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:35.669185 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 11:32:35.669321 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 11:32:35.669693 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 11:32:35.670025 14291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.654852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40752 (local address 127.23.153.61:35839)
0929 11:32:35.655102 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:35.655109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:35.655202 (+ 93us) server_negotiation.cc:407] Connection header received
0929 11:32:35.655409 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:35.655415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:35.655506 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:35.655605 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:35.656633 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:35.657462 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:35.658726 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:35.659279 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:35.659407 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:35.663325 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:35.663336 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:35.663342 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:35.663411 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:35.666655 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:35.667960 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:35.667966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:35.667969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:35.668146 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:35.668588 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:35.668591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:35.668593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:35.668894 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:35.669043 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:35.669440 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 11:32:35.669679 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 11:32:35.671188 14212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:32:35.672286 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.671886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:60976)
0929 11:32:35.672051 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.672136 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 11:32:35.673182 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.672845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:60982)
0929 11:32:35.672987 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.673062 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 11:32:35.673370 14212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:35425: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
I20260929 11:32:35.696761 14286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } attempt: 2
I20260929 11:32:35.698047 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.697538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:60996)
0929 11:32:35.697775 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.697870 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 11:32:35.699637 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.698868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:32778)
0929 11:32:35.699259 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.699415 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 11:32:35.700088 14286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:35425: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
I20260929 11:32:35.724613 14212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } attempt: 1
I20260929 11:32:35.726272 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.725518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:32792)
0929 11:32:35.725905 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.726051 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260929 11:32:35.727392 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.726982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:32808)
0929 11:32:35.727183 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.727262 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260929 11:32:35.727612 14212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:35425: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
I20260929 11:32:35.791455 14286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } attempt: 3
I20260929 11:32:35.792802 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.792240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:32810)
0929 11:32:35.792511 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.792622 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 11:32:35.794030 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.793555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:32812)
0929 11:32:35.793792 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.793878 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260929 11:32:35.794256 14286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:35425: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
I20260929 11:32:35.795811 14212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } attempt: 2
I20260929 11:32:35.797137 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.796528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:32826)
0929 11:32:35.796821 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.796923 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 11:32:35.798166 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.797813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:32834)
0929 11:32:35.797965 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.798047 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 11:32:35.798386 14212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:35425: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
I20260929 11:32:35.904639 14212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } attempt: 3
I20260929 11:32:35.905833 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.905342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:32836)
0929 11:32:35.905555 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.905663 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 11:32:35.906890 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.906478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:32852)
0929 11:32:35.906701 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.906778 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260929 11:32:35.907114 14212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:35425: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
I20260929 11:32:35.937510 14286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } attempt: 4
I20260929 11:32:35.938624 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.938170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:32868)
0929 11:32:35.938383 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.938474 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 11:32:35.939824 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:35.939464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:32884)
0929 11:32:35.939621 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:35.939708 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 11:32:35.940062 14286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:35425: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
W20260929 11:32:35.967275 14294 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 11:32:35.967646 14294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:35.967721 14294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:35.967806 14294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:35.975967 14294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:35.976096 14294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:35.976155 14294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:35.976205 14294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:35.976269 14294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:35.987118 14294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:40527,127.23.153.61:35839,127.23.153.60:35425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.60.keytab
--principal=kudu/127.23.153.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.60:35425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.153.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:35.988503 14294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:35.989959 14294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:36.005764 14302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:36.005792 14300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:36.005717 14299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:36.008020 14294 server_base.cc:1034] running on GCE node
I20260929 11:32:36.008980 14294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:36.010607 14294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:36.011881 14294 hybrid_clock.cc:648] HybridClock initialized: now 1790681556011817 us; error 45 us; skew 500 ppm
I20260929 11:32:36.012432 14294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:36 dist-test-slave-qk05 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681556, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:36.023103 14294 init.cc:377] Logged in from keytab as kudu/127.23.153.60@KRBTEST.COM (short username kudu)
I20260929 11:32:36.024829 14294 webserver.cc:460] Webserver started at http://127.23.153.60:35351/ using document root <none> and password file <none>
I20260929 11:32:36.025635 14294 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:36.025777 14294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:36.026132 14294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:36.029515 14294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-2/data/instance:
uuid: "8a3cf78028eb4531a8f0c3317dbe8660"
format_stamp: "Formatted at 2026-09-29 11:32:36 on dist-test-slave-qk05"
server_key: "4ec5922f5e65aa37dba319001a920384"
server_key_iv: "749de754e87da34cf68ece4235c077ba"
server_key_version: "encryptionkey@0"
I20260929 11:32:36.030431 14294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-2/wal/instance:
uuid: "8a3cf78028eb4531a8f0c3317dbe8660"
format_stamp: "Formatted at 2026-09-29 11:32:36 on dist-test-slave-qk05"
server_key: "4ec5922f5e65aa37dba319001a920384"
server_key_iv: "749de754e87da34cf68ece4235c077ba"
server_key_version: "encryptionkey@0"
I20260929 11:32:36.036844 14294 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 11:32:36.041656 14309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:36.043355 14294 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 11:32:36.043572 14294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-2/wal
uuid: "8a3cf78028eb4531a8f0c3317dbe8660"
format_stamp: "Formatted at 2026-09-29 11:32:36 on dist-test-slave-qk05"
server_key: "4ec5922f5e65aa37dba319001a920384"
server_key_iv: "749de754e87da34cf68ece4235c077ba"
server_key_version: "encryptionkey@0"
I20260929 11:32:36.043817 14294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:36.084372 14212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } attempt: 4
I20260929 11:32:36.085680 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:36.085185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:32886)
0929 11:32:36.085409 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:36.085509 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 11:32:36.086838 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:36.086475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:32902)
0929 11:32:36.086634 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:36.086708 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 11:32:36.087088 14212 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:35425: Network error: Client connection negotiation failed: client connection to 127.23.153.60:35425: connect: Connection refused (error 111)
I20260929 11:32:36.102700 14294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:36.107883 14294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:36.108196 14294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:36.128422 14294 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.60:35425
I20260929 11:32:36.128530 14360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.60:35425 every 8 connection(s)
I20260929 11:32:36.129997 14294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-2/data/info.pb
I20260929 11:32:36.132522 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 14294
I20260929 11:32:36.132718 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790681181154781-24164-0/minicluster-data/master-2/wal/instance
I20260929 11:32:36.133270 24164 external_mini_cluster.cc:1428] Setting key 64efb805744f801df189332a30b829ae
I20260929 11:32:36.135808 14361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:36.146819 14361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:32:36 dist-test-slave-qk05 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.60@KRBTEST.COM
Sep 29 11:32:36 dist-test-slave-qk05 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681556, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.60@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:32:36.159634 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:36.135537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.23.153.60:35425)
0929 11:32:36.136654 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.136667 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.136711 (+ 44us) server_negotiation.cc:407] Connection header received
0929 11:32:36.139350 (+ 2639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.139384 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.139920 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.140393 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.142071 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.143666 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.145090 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.145893 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.146085 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:36.151446 (+ 5361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:36.151458 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:36.151478 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:36.151528 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:36.154602 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.155491 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.155498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.155506 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.155704 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.156084 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.156088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.156090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.156744 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:36.157062 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:36.157586 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 11:32:36.158067 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":571,"threads_started":1}
I20260929 11:32:36.167196 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:36.150219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:40527 (local address 127.0.0.1:55694)
0929 11:32:36.150977 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:36.151065 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:36.151242 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:36.151996 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:36.152005 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:36.152780 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:36.153121 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:36.153132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.154260 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.154264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:36.155450 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:36.155465 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.156241 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.156248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:36.156416 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:36.157128 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:36.157152 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:36.160571 (+ 3419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:36.164237 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:36.164253 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:36.164260 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:36.164772 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:36.165197 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:36.165202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:36.165204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:36.165341 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:36.165969 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:36.165979 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:36.166181 (+ 202us) client_negotiation.cc:769] Sending connection context
0929 11:32:36.166345 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 11:32:36.166718 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":315,"threads_started":1}
I20260929 11:32:36.167240 14366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:36.150450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55694 (local address 127.23.153.62:40527)
0929 11:32:36.151671 (+ 1221us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.151675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.151692 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:32:36.151743 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.151746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.151806 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.151907 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.153312 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.154081 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.155626 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.156039 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.156159 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:36.160842 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:36.160851 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:36.160855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:36.160905 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:36.164073 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.164912 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.164916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.164919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.165080 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.165461 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.165465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.165468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.165824 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:36.166006 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:36.166511 (+ 505us) server_negotiation.cc:299] Negotiation successful
0929 11:32:36.166740 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1119,"thread_start_us":965,"threads_started":1}
I20260929 11:32:36.169202 14361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:32:36 dist-test-slave-qk05 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
Sep 29 11:32:36 dist-test-slave-qk05 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681556, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.60@KRBTEST.COM for kudu/127.23.153.61@KRBTEST.COM
I20260929 11:32:36.180691 14367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:36.163200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55704 (local address 127.23.153.62:40527)
0929 11:32:36.163851 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.163856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.163877 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:32:36.163931 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.163934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.163996 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.164116 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.165792 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.167284 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.168864 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.169521 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.169665 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:36.174682 (+ 5017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:36.174688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:36.174691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:36.174727 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:36.177825 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.178593 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.178597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.178599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.178744 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.179172 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.179176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.179178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.179477 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:36.179749 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:36.180085 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 11:32:36.180344 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":260,"threads_started":1}
I20260929 11:32:36.184971 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:36.170062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:35839 (local address 127.0.0.1:40754)
0929 11:32:36.170238 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:36.170257 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:36.170350 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:36.170662 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:36.170665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:36.171060 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:36.171218 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:36.171229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.172595 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.172599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:36.173490 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:36.173506 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.174316 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.174323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:36.174533 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:36.175124 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:36.175142 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:36.177569 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:36.181095 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:36.181099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:36.181103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:36.182066 (+ 963us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:36.182741 (+ 675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:36.182749 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:36.182753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:36.183083 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:36.183934 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:36.183938 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:36.184107 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 11:32:36.184245 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 11:32:36.184594 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 11:32:36.185024 14291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:36.170136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40754 (local address 127.23.153.61:35839)
0929 11:32:36.170331 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.170336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.170357 (+ 21us) server_negotiation.cc:407] Connection header received
0929 11:32:36.170463 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.170467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.170530 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.170624 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.171368 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.172432 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.173675 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.174111 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.174213 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:36.177845 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:36.177853 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:36.177857 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:36.177906 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:36.180921 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.182308 (+ 1387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.182314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.182317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.182516 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.183319 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.183326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.183330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.183790 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:36.183977 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:36.184405 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 11:32:36.184644 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 11:32:36.186025 14361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:32:36 dist-test-slave-qk05 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.61@KRBTEST.COM
Sep 29 11:32:36 dist-test-slave-qk05 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681556, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.60@KRBTEST.COM for kudu/127.23.153.60@KRBTEST.COM
I20260929 11:32:36.200393 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:36.186803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.23.153.60:35425)
0929 11:32:36.186966 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.186972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.187155 (+ 183us) server_negotiation.cc:407] Connection header received
0929 11:32:36.187300 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.187305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.187366 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.187438 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.188138 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.188847 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.189969 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.190371 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.190478 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:36.194008 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:36.194014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:36.194018 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:36.194054 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:36.197440 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.198281 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.198291 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.198294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.198474 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.198900 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.198905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.198908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.199252 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:36.199387 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:36.199742 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 11:32:36.199996 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":51}
I20260929 11:32:36.200495 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:36.182934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.23.153.61:35839)
0929 11:32:36.184389 (+ 1455us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.184393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.184410 (+ 17us) server_negotiation.cc:407] Connection header received
0929 11:32:36.184450 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.184454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.184508 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.184598 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.186157 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.187279 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.188553 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.188994 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.189121 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:36.193972 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:36.193980 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:36.193985 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:36.194055 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:36.197218 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.198195 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.198201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.198205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.198384 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.198868 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.198873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.198876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.199281 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:36.199550 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:36.199862 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 11:32:36.200102 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1346,"thread_start_us":1183,"threads_started":1}
I20260929 11:32:36.200495 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:36.186896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:32914)
0929 11:32:36.187067 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:36.187086 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:36.187205 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:36.187466 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:36.187470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:36.187871 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:36.188006 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:36.188016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.188967 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.188972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:36.189834 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:36.189846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.190524 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.190529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:36.190742 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:36.191400 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:36.191419 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:36.193820 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:36.197608 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:36.197612 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:36.197614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:36.198127 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:36.198629 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:36.198634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:36.198637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:36.198803 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:36.199362 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:36.199367 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:36.199506 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 11:32:36.199645 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 11:32:36.200186 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":191}
I20260929 11:32:36.206043 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:36.207093 14361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660: Bootstrap starting.
I20260929 11:32:36.211294 14361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:36.212471 14361 log.cc:824] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:36.210213 14187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b83f991d5436402697bca98b46050569"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55704
I20260929 11:32:36.215618 14361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660: No bootstrap required, opened a new log
I20260929 11:32:36.224347 14361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f991d5436402697bca98b46050569" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } } peers { permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } } peers { permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } }
I20260929 11:32:36.225163 14361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3cf78028eb4531a8f0c3317dbe8660, State: Initialized, Role: FOLLOWER
I20260929 11:32:36.225792 14361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f991d5436402697bca98b46050569" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } } peers { permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } } peers { permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } }
I20260929 11:32:36.226984 14370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f991d5436402697bca98b46050569" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } } peers { permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } } peers { permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } } }
I20260929 11:32:36.227536 14370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:32:36.228681 14361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:36.239354 14286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } attempt: 5
W20260929 11:32:36.239760 14381 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:32:36.239898 14381 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:32:36.237151 14294 master_runner.cc:186] Error getting master registration for 127.23.153.62:40527: OK, instance_id { permanent_uuid: "b83f991d5436402697bca98b46050569" instance_seqno: 1790681554927935 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 11:32:36.251008 14294 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 29 11:32:36 dist-test-slave-qk05 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681555, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.61@KRBTEST.COM for kudu/127.23.153.60@KRBTEST.COM
I20260929 11:32:36.260391 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:36.240308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:32926)
0929 11:32:36.240752 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:36.240775 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:36.240916 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:36.242050 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:36.242055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:36.242780 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:36.243071 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:36.243088 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.245156 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.245163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:36.246657 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:36.246682 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.249197 (+ 2515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.249205 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:36.249504 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:36.250278 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:36.250306 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:36.253141 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:36.257626 (+ 4485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:36.257633 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:36.257637 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:36.258090 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:36.258513 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:36.258519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:36.258523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:36.258671 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:36.259373 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:36.259378 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:36.259537 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 11:32:36.259653 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 11:32:36.260008 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306}
I20260929 11:32:36.260398 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:36.241314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.23.153.60:35425)
0929 11:32:36.241533 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.241539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.241558 (+ 19us) server_negotiation.cc:407] Connection header received
0929 11:32:36.241630 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.241635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.241731 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.241841 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.243431 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.244853 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.247526 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.248981 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.249242 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:36.253415 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:36.253420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:36.253423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:36.253461 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:36.257402 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.258249 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.258254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.258258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.258391 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.258800 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.258806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.258809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.259249 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:36.259412 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:36.259794 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 11:32:36.260022 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 11:32:36.266327 14286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d: Bootstrap starting.
I20260929 11:32:36.270661 14286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:36.272119 14286 log.cc:824] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:36.276297 14286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d: No bootstrap required, opened a new log
I20260929 11:32:36.288271 14286 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f991d5436402697bca98b46050569" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } } peers { permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } } peers { permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } }
I20260929 11:32:36.288762 14286 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3beb5c821b459281fb9b769517017d, State: Initialized, Role: FOLLOWER
I20260929 11:32:36.289268 14286 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f991d5436402697bca98b46050569" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } } peers { permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } } peers { permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } }
I20260929 11:32:36.290331 14384 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f991d5436402697bca98b46050569" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } } peers { permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } } peers { permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } } }
I20260929 11:32:36.290817 14384 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:32:36.291800 14286 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d [sys.catalog]: configured and running, proceeding with master startup.
W20260929 11:32:36.303162 14395 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:32:36.303476 14395 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:32:36.299602 14219 master_runner.cc:186] Error getting master registration for 127.23.153.62:40527: OK, instance_id { permanent_uuid: "b83f991d5436402697bca98b46050569" instance_seqno: 1790681554927935 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 11:32:36.311619 14219 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 11:32:36.361094 14315 tablet.cc:2369] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:32:36.364297 14212 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } attempt: 5
Sep 29 11:32:36 dist-test-slave-qk05 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681554, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for kudu/127.23.153.60@KRBTEST.COM
I20260929 11:32:36.383697 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:36.365209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:35425 (local address 127.0.0.1:32932)
0929 11:32:36.365521 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:36.365545 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:36.365703 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:36.366115 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:36.366120 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:36.366647 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:36.366870 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:36.366884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.368564 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.368571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:36.370180 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:36.370205 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.371572 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.371579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:36.371790 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:36.372534 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:36.372560 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:36.375720 (+ 3160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:36.380331 (+ 4611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:36.380338 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:36.380342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:36.380939 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:36.381497 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:36.381503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:36.381508 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:36.381694 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:36.382445 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:36.382451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:36.382623 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 11:32:36.382779 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 11:32:36.383231 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260929 11:32:36.384743 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:36.365293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.23.153.60:35425)
0929 11:32:36.365586 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.365604 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.365629 (+ 25us) server_negotiation.cc:407] Connection header received
0929 11:32:36.365856 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.365863 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.365953 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.366067 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.367120 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.368350 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.370438 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.371178 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.371325 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:36.376031 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:36.376040 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:36.376044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:36.376094 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:36.380035 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.381135 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.381142 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.381146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.381345 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.381833 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.381838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.381841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.382214 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:36.382367 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:36.384118 (+ 1751us) server_negotiation.cc:299] Negotiation successful
0929 11:32:36.384385 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260929 11:32:36.391108 14212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569: Bootstrap starting.
I20260929 11:32:36.395870 14212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:36.397329 14212 log.cc:824] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:36.401551 14212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569: No bootstrap required, opened a new log
I20260929 11:32:36.410051 14212 raft_consensus.cc:348] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f991d5436402697bca98b46050569" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } } peers { permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } } peers { permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } }
I20260929 11:32:36.410633 14212 raft_consensus.cc:729] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83f991d5436402697bca98b46050569, State: Initialized, Role: FOLLOWER
I20260929 11:32:36.411247 14212 consensus_queue.cc:260] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f991d5436402697bca98b46050569" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } } peers { permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } } peers { permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } }
I20260929 11:32:36.412665 14398 sys_catalog.cc:455] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f991d5436402697bca98b46050569" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } } peers { permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } } peers { permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } } }
I20260929 11:32:36.413443 14398 sys_catalog.cc:458] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:32:36.413878 14212 sys_catalog.cc:564] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:36.413939 14187 raft_consensus.cc:482] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 11:32:36.414146 14187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [term 0 FOLLOWER]: Advancing to term 1
W20260929 11:32:36.416985 14166 tablet.cc:2369] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:32:36.417840 14187 raft_consensus.cc:504] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f991d5436402697bca98b46050569" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } } peers { permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } } peers { permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } }
I20260929 11:32:36.420486 14261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b83f991d5436402697bca98b46050569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ad3beb5c821b459281fb9b769517017d"
I20260929 11:32:36.421054 14261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:36.425077 14261 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b83f991d5436402697bca98b46050569 in term 1.
I20260929 11:32:36.426095 14161 leader_election.cc:302] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad3beb5c821b459281fb9b769517017d, b83f991d5436402697bca98b46050569; no voters:
I20260929 11:32:36.428050 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b83f991d5436402697bca98b46050569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8a3cf78028eb4531a8f0c3317dbe8660"
I20260929 11:32:36.428668 14336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:36.432946 14336 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b83f991d5436402697bca98b46050569 in term 1.
I20260929 11:32:36.433934 14187 leader_election.cc:288] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [CANDIDATE]: Term 1 election: Requested vote from peers ad3beb5c821b459281fb9b769517017d (127.23.153.61:35839), 8a3cf78028eb4531a8f0c3317dbe8660 (127.23.153.60:35425)
I20260929 11:32:36.441215 14398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:36.451386 14398 raft_consensus.cc:686] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [term 1 LEADER]: Becoming Leader. State: Replica: b83f991d5436402697bca98b46050569, State: Running, Role: LEADER
I20260929 11:32:36.452519 14398 consensus_queue.cc:237] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f991d5436402697bca98b46050569" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } } peers { permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } } peers { permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } }
I20260929 11:32:36.484264 14404 sys_catalog.cc:455] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b83f991d5436402697bca98b46050569. Latest consensus state: current_term: 1 leader_uuid: "b83f991d5436402697bca98b46050569" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f991d5436402697bca98b46050569" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } } peers { permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } } peers { permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } } }
I20260929 11:32:36.484733 14404 sys_catalog.cc:458] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:36.495816 14418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260929 11:32:36.497709 14240 tablet.cc:2369] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:32:36.508793 14418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 11:32:36.541355 14336 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660 [term 1 FOLLOWER]: Refusing update from remote peer b83f991d5436402697bca98b46050569: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:32:36.543076 14404 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:32:36.546806 14364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:36.471307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.23.153.60:35425)
0929 11:32:36.471780 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.471791 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.486547 (+ 14756us) server_negotiation.cc:407] Connection header received
0929 11:32:36.487773 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.487780 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.487876 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.487956 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.494876 (+ 6920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.496099 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.511358 (+ 15259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.512237 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.512359 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:36.526259 (+ 13900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:36.526267 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:36.526273 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:36.526328 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:36.530116 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.539781 (+ 9665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.539788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.539792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.540015 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.542044 (+ 2029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.542051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.542054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.542508 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:36.542782 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:36.546053 (+ 3271us) server_negotiation.cc:299] Negotiation successful
0929 11:32:36.546371 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 11:32:36.548964 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:36.469105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.23.153.61:35839)
0929 11:32:36.469298 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.469305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.486047 (+ 16742us) server_negotiation.cc:407] Connection header received
0929 11:32:36.490389 (+ 4342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.490395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.490487 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.490560 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.498424 (+ 7864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.499713 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.514255 (+ 14542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.515046 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.515172 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:36.531532 (+ 16360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:36.531541 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:36.531548 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:36.531604 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:36.535617 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.540636 (+ 5019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.540643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.540647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.540834 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.543345 (+ 2511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.543352 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.543355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.543788 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:36.544049 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:36.548294 (+ 4245us) server_negotiation.cc:299] Negotiation successful
0929 11:32:36.548598 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 11:32:36.551782 14261 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d [term 1 FOLLOWER]: Refusing update from remote peer b83f991d5436402697bca98b46050569: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:32:36.552892 14367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:36.477642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.23.153.62:40527)
0929 11:32:36.477876 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.477883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.487644 (+ 9761us) server_negotiation.cc:407] Connection header received
0929 11:32:36.491333 (+ 3689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.491339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.491427 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.491523 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.502323 (+ 10800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.503539 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.516684 (+ 13145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:36.517529 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:36.517652 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:36.533959 (+ 16307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:36.533967 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:36.533972 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:36.534027 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:36.537851 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.541349 (+ 3498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.541356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.541360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.541532 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:36.544549 (+ 3017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:36.544555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:36.544559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:36.545004 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:36.545296 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:36.552213 (+ 6917us) server_negotiation.cc:299] Negotiation successful
0929 11:32:36.552510 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:32:36.552780 14404 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:32:36.565833 14370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b83f991d5436402697bca98b46050569. Latest consensus state: current_term: 1 leader_uuid: "b83f991d5436402697bca98b46050569" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f991d5436402697bca98b46050569" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } } peers { permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } } peers { permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } } }
I20260929 11:32:36.566186 14370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:32:36.577725 14421 mvcc.cc:205] Tried to move back new op lower bound from 7334631655541444608 to 7334631655251300352. Current Snapshot: MvccSnapshot[applied={T|T < 7334631655541444608}]
I20260929 11:32:36.592672 14384 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b83f991d5436402697bca98b46050569. Latest consensus state: current_term: 1 leader_uuid: "b83f991d5436402697bca98b46050569" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f991d5436402697bca98b46050569" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } } peers { permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } } peers { permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } } }
I20260929 11:32:36.593043 14384 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:32:36.597640 14423 mvcc.cc:205] Tried to move back new op lower bound from 7334631655541444608 to 7334631655251300352. Current Snapshot: MvccSnapshot[applied={T|T < 7334631655541444608}]
I20260929 11:32:36.608778 14384 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b83f991d5436402697bca98b46050569" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f991d5436402697bca98b46050569" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } } peers { permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } } peers { permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } } }
I20260929 11:32:36.609040 14384 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:32:36.613425 14370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b83f991d5436402697bca98b46050569" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f991d5436402697bca98b46050569" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } } peers { permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } } peers { permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } } }
I20260929 11:32:36.613749 14370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:32:36.614323 14404 sys_catalog.cc:455] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b83f991d5436402697bca98b46050569" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f991d5436402697bca98b46050569" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } } peers { permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } } peers { permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } } }
I20260929 11:32:36.614614 14404 sys_catalog.cc:458] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:36.614925 14404 sys_catalog.cc:455] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b83f991d5436402697bca98b46050569" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f991d5436402697bca98b46050569" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 40527 } } peers { permanent_uuid: "ad3beb5c821b459281fb9b769517017d" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 35839 } } peers { permanent_uuid: "8a3cf78028eb4531a8f0c3317dbe8660" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 35425 } } }
I20260929 11:32:36.615187 14404 sys_catalog.cc:458] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:36.615597 14418 catalog_manager.cc:1347] Generated new cluster ID: 34870318f5424fc98d0e7e0f66254e85
I20260929 11:32:36.615694 14418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:36.643889 14418 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:36.646016 14418 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:36.670763 14418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b83f991d5436402697bca98b46050569: Generated new TSK 0
I20260929 11:32:36.672205 14418 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 11:32:36.829856 14367 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:36.794612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55730 (local address 127.23.153.62:40527)
0929 11:32:36.794818 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.794824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.825069 (+ 30245us) server_negotiation.cc:407] Connection header received
0929 11:32:36.828049 (+ 2980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.828054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.828164 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.828234 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.829557 (+ 1323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55730: BlockingRecv error: recv got EOF from 127.0.0.1:55730 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 11:32:36.832803 14364 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:36.807394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32948 (local address 127.23.153.60:35425)
0929 11:32:36.807645 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.807652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.830908 (+ 23256us) server_negotiation.cc:407] Connection header received
0929 11:32:36.831224 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.831230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.831318 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.831392 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.832526 (+ 1134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32948: BlockingRecv error: recv got EOF from 127.0.0.1:32948 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 11:32:36.840369 14368 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:36.796817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.23.153.61:35839)
0929 11:32:36.797016 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.797023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.827617 (+ 30594us) server_negotiation.cc:407] Connection header received
0929 11:32:36.827780 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.827786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.827902 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.828018 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.840102 (+ 12084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40792: BlockingRecv error: recv got EOF from 127.0.0.1:40792 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 11:32:36.880767 14368 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:36.867471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.23.153.61:35839)
0929 11:32:36.867672 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.867678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.878573 (+ 10895us) server_negotiation.cc:407] Connection header received
0929 11:32:36.879170 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.879176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.879304 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.879378 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.880519 (+ 1141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40796: BlockingRecv error: recv got EOF from 127.0.0.1:40796 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 11:32:36.891435 14368 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:36.889403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40812 (local address 127.23.153.61:35839)
0929 11:32:36.889619 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.889626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.889741 (+ 115us) server_negotiation.cc:407] Connection header received
0929 11:32:36.890040 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.890047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.890147 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.890263 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.891162 (+ 899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40812: BlockingRecv error: recv got EOF from 127.0.0.1:40812 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 11:32:36.899401 14368 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:36.897538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40828 (local address 127.23.153.61:35839)
0929 11:32:36.897769 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.897776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.897960 (+ 184us) server_negotiation.cc:407] Connection header received
0929 11:32:36.898181 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.898185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.898262 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.898352 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.899166 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40828: BlockingRecv error: recv got EOF from 127.0.0.1:40828 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 11:32:36.913172 14368 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:36.906933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40842 (local address 127.23.153.61:35839)
0929 11:32:36.907259 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.907266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.911726 (+ 4460us) server_negotiation.cc:407] Connection header received
0929 11:32:36.911928 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.911932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.912015 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.912104 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.912884 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40842: BlockingRecv error: recv got EOF from 127.0.0.1:40842 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 11:32:36.927929 14368 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:36.925996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.23.153.61:35839)
0929 11:32:36.926203 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.926210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.926489 (+ 279us) server_negotiation.cc:407] Connection header received
0929 11:32:36.926668 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.926673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.926789 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.926924 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.927704 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40850: BlockingRecv error: recv got EOF from 127.0.0.1:40850 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260929 11:32:36.952183 14368 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:36.949662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40858 (local address 127.23.153.61:35839)
0929 11:32:36.949878 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.949884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.950504 (+ 620us) server_negotiation.cc:407] Connection header received
0929 11:32:36.950715 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.950721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.950824 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.950946 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.951873 (+ 927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40858: BlockingRecv error: recv got EOF from 127.0.0.1:40858 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 11:32:36.990059 14368 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:36.988022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.23.153.61:35839)
0929 11:32:36.988322 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:36.988328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:36.988559 (+ 231us) server_negotiation.cc:407] Connection header received
0929 11:32:36.988745 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:36.988751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:36.988858 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:36.988997 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:36.989819 (+ 822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40872: BlockingRecv error: recv got EOF from 127.0.0.1:40872 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260929 11:32:37.060087 14368 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:37.058004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40882 (local address 127.23.153.61:35839)
0929 11:32:37.058231 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:37.058238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:37.058552 (+ 314us) server_negotiation.cc:407] Connection header received
0929 11:32:37.058717 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:37.058723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:37.058829 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:37.058953 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:37.059844 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40882: BlockingRecv error: recv got EOF from 127.0.0.1:40882 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260929 11:32:37.193313 14368 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:37.191506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.23.153.61:35839)
0929 11:32:37.191738 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:37.191745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:37.191953 (+ 208us) server_negotiation.cc:407] Connection header received
0929 11:32:37.192171 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:37.192175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:37.192254 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:37.192339 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:37.193089 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40886: BlockingRecv error: recv got EOF from 127.0.0.1:40886 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 11:32:37.242074 14381 catalog_manager.cc:1259] Loaded cluster ID: 34870318f5424fc98d0e7e0f66254e85
I20260929 11:32:37.242190 14381 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660: loading cluster ID for follower catalog manager: success
I20260929 11:32:37.245477 14381 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660: acquiring CA information for follower catalog manager: success
I20260929 11:32:37.248075 14381 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8a3cf78028eb4531a8f0c3317dbe8660: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 11:32:37.305577 14395 catalog_manager.cc:1259] Loaded cluster ID: 34870318f5424fc98d0e7e0f66254e85
I20260929 11:32:37.305712 14395 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d: loading cluster ID for follower catalog manager: success
I20260929 11:32:37.308319 14395 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d: acquiring CA information for follower catalog manager: success
I20260929 11:32:37.310307 14395 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ad3beb5c821b459281fb9b769517017d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 11:32:37.461553 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:37.454749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55386 (local address 127.23.153.61:35839)
0929 11:32:37.454965 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:37.454971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:37.455203 (+ 232us) server_negotiation.cc:407] Connection header received
0929 11:32:37.455375 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:37.455379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:37.455546 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:37.455651 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:32:37.456142 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:37.456908 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:37.458430 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:37.458882 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:37.459003 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:37.459892 (+ 889us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:32:37.460606 (+ 714us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:32:37.460762 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:37.460945 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 11:32:37.461063 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 11:32:37.477164 14493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:37.470189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33318 (local address 127.23.153.60:35425)
0929 11:32:37.470626 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:37.470631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:37.470688 (+ 57us) server_negotiation.cc:407] Connection header received
0929 11:32:37.470956 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:37.470961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:37.471303 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:37.471469 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:32:37.471939 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:37.472685 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:37.474666 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:37.475135 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:37.475278 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:37.475880 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:32:37.476469 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:32:37.476613 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:37.476726 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 11:32:37.476820 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":240,"threads_started":1}
I20260929 11:32:37.503154 14493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:37.484756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.23.153.60:35425)
0929 11:32:37.484933 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:37.484939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:37.496822 (+ 11883us) server_negotiation.cc:407] Connection header received
0929 11:32:37.497014 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:37.497019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:37.497122 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:37.497216 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:32:37.497876 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:37.498696 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:37.501078 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:37.501737 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:37.501839 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:37.502365 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:32:37.502487 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:32:37.502565 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:37.502715 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 11:32:37.502804 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 11:32:37.503988 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:37.484245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.23.153.62:40527)
0929 11:32:37.494623 (+ 10378us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:37.494633 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:37.495480 (+ 847us) server_negotiation.cc:407] Connection header received
0929 11:32:37.495794 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:37.495804 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:37.496039 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:37.496249 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:32:37.496938 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:37.498049 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:37.501020 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:37.501502 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:37.501638 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:37.502520 (+ 882us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:32:37.503001 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:32:37.503325 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:37.503576 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 11:32:37.503686 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10184,"thread_start_us":10437,"threads_started":1}
I20260929 11:32:37.504860 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:37.484547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55394 (local address 127.23.153.61:35839)
0929 11:32:37.484741 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:37.484748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:37.496336 (+ 11588us) server_negotiation.cc:407] Connection header received
0929 11:32:37.497147 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:37.497153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:37.497239 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:37.497319 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:32:37.497956 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:37.499444 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:37.501969 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:37.502754 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:37.502903 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:37.503768 (+ 865us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:32:37.503948 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:32:37.504084 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:37.504502 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 11:32:37.504598 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 11:32:37.511667 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 14145
I20260929 11:32:37.513013 14207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:37.633303 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 14145
W20260929 11:32:37.653568 14236 connection.cc:535] server connection from 127.0.0.1:40752 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 11:32:37.653908 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 14219
I20260929 11:32:37.655081 14281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:37.774417 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 14219
I20260929 11:32:37.792634 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 14294
I20260929 11:32:37.793826 14356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:37.904340 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 14294
2026-09-29T11:32:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4797 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 11:32:38 dist-test-slave-qk05 krb5kdc[14507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 11:32:38 dist-test-slave-qk05 krb5kdc[14507](info): set up 2 sockets
Sep 29 11:32:38 dist-test-slave-qk05 krb5kdc[14507](info): commencing operation
krb5kdc: starting...
W20260929 11:32:38.971398 24164 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/test-user.keytab.
Sep 29 11:32:39 dist-test-slave-qk05 krb5kdc[14507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T11:32:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T11:32:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.153.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
Entry for principal kudu/127.23.153.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab.
I20260929 11:32:39.251431 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:39025
--webserver_interface=127.23.153.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:39025,127.23.153.61:38983,127.23.153.60:40977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 11:32:39.609620 14520 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 11:32:39.610287 14520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:39.610437 14520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:39.610618 14520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:39.618496 14520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:39.618614 14520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:39.618674 14520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:39.618744 14520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:39.618818 14520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:39.629205 14520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:39025,127.23.153.61:38983,127.23.153.60:40977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.62.keytab
--principal=kudu/127.23.153.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.62:39025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.153.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:39.630748 14520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:39.632303 14520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:39.644768 14525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:39.644827 14528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:39.644956 14526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:39.646313 14520 server_base.cc:1034] running on GCE node
I20260929 11:32:39.647769 14520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:39.649636 14520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:39.650998 14520 hybrid_clock.cc:648] HybridClock initialized: now 1790681559650874 us; error 51 us; skew 500 ppm
I20260929 11:32:39.651664 14520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:39 dist-test-slave-qk05 krb5kdc[14507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681559, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:39.655702 14520 init.cc:377] Logged in from keytab as kudu/127.23.153.62@KRBTEST.COM (short username kudu)
I20260929 11:32:39.657564 14520 webserver.cc:460] Webserver started at http://127.23.153.62:40897/ using document root <none> and password file <none>
I20260929 11:32:39.658356 14520 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:39.658485 14520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:39.658819 14520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:39.662194 14520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-0/data/instance:
uuid: "7d75014d97084589a5e483793a728b61"
format_stamp: "Formatted at 2026-09-29 11:32:39 on dist-test-slave-qk05"
server_key: "e624dff0bc2760e78c49875872400d75"
server_key_iv: "c838969341e4c545f2a4a2765612dbb5"
server_key_version: "encryptionkey@0"
I20260929 11:32:39.663204 14520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-0/wal/instance:
uuid: "7d75014d97084589a5e483793a728b61"
format_stamp: "Formatted at 2026-09-29 11:32:39 on dist-test-slave-qk05"
server_key: "e624dff0bc2760e78c49875872400d75"
server_key_iv: "c838969341e4c545f2a4a2765612dbb5"
server_key_version: "encryptionkey@0"
I20260929 11:32:39.669354 14520 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:32:39.673633 14535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:39.675552 14520 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:32:39.675770 14520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-0/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-0/wal
uuid: "7d75014d97084589a5e483793a728b61"
format_stamp: "Formatted at 2026-09-29 11:32:39 on dist-test-slave-qk05"
server_key: "e624dff0bc2760e78c49875872400d75"
server_key_iv: "c838969341e4c545f2a4a2765612dbb5"
server_key_version: "encryptionkey@0"
I20260929 11:32:39.676021 14520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:39.698171 14520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:39.704248 14520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:39.704638 14520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:39.727326 14586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.62:39025 every 8 connection(s)
I20260929 11:32:39.727486 14520 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.62:39025
I20260929 11:32:39.729350 14520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-0/data/info.pb
I20260929 11:32:39.730034 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 14520
I20260929 11:32:39.730270 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-0/wal/instance
I20260929 11:32:39.730918 24164 external_mini_cluster.cc:1428] Setting key cc0ef5da960d4acda663ad72586a275f
I20260929 11:32:39.734848 14587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:39.742329 14587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.153.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.61@KRBTEST.COM" created.
Sep 29 11:32:39 dist-test-slave-qk05 krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681559, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:32:39.769979 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:39.747617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46676 (local address 127.23.153.62:39025)
0929 11:32:39.748359 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:39.748368 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:39.748427 (+ 59us) server_negotiation.cc:407] Connection header received
0929 11:32:39.749890 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:39.749899 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:39.750302 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:39.750572 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:39.752128 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:39.754153 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:39.755965 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:39.756659 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:39.756876 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:39.761786 (+ 4910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:39.761796 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:39.761812 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:39.761852 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:39.764912 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:39.766053 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:39.766072 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:39.766075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:39.766267 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:39.766784 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:39.766789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:39.766792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:39.767597 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:39.767848 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:39.768473 (+ 625us) server_negotiation.cc:299] Negotiation successful
0929 11:32:39.768922 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":190,"server-negotiator.queue_time_us":518,"thread_start_us":338,"threads_started":1}
I20260929 11:32:39.770510 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:39.746067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:39025 (local address 127.0.0.1:46676)
0929 11:32:39.747006 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:39.747097 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:39.749380 (+ 2283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:39.750632 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:39.750638 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:39.751470 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:39.751852 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:39.751874 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:39.754423 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:39.754428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:39.755758 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:39.755777 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:39.756957 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:39.756965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:39.757249 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:39.757958 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:39.757989 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:39.761421 (+ 3432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:39.765153 (+ 3732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:39.765169 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:39.765185 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:39.765862 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:39.766400 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:39.766405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:39.766409 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:39.766657 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:39.767794 (+ 1137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:39.767801 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:39.768024 (+ 223us) client_negotiation.cc:769] Sending connection context
0929 11:32:39.768261 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 11:32:39.768746 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":42,"thread_start_us":403,"threads_started":1}
I20260929 11:32:39.775853 14587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:32:39.777565 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:39.777004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:38983 (local address 127.0.0.1:54280)
0929 11:32:39.777229 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:39.777410 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260929 11:32:39.777944 14537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.153.61:38983: connect: Connection refused (error 111)
I20260929 11:32:39.779950 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:39.779474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:38983 (local address 127.0.0.1:54288)
0929 11:32:39.779676 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:39.779766 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 11:32:39.780201 14587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.61:38983: Network error: Client connection negotiation failed: client connection to 127.23.153.61:38983: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.61.keytab.
Entry for principal kudu/127.23.153.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.61.keytab.
I20260929 11:32:39.795650 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.61:38983
--webserver_interface=127.23.153.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.61.keytab
--principal=kudu/127.23.153.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:39025,127.23.153.61:38983,127.23.153.60:40977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:32:39.829617 14587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } attempt: 1
I20260929 11:32:39.830835 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:39.830346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:38983 (local address 127.0.0.1:54294)
0929 11:32:39.830545 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:39.830692 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 11:32:39.832022 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:39.831659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:38983 (local address 127.0.0.1:54304)
0929 11:32:39.831793 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:39.831870 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 11:32:39.832228 14587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.61:38983: Network error: Client connection negotiation failed: client connection to 127.23.153.61:38983: connect: Connection refused (error 111)
I20260929 11:32:39.900502 14587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } attempt: 2
I20260929 11:32:39.901757 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:39.901252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:38983 (local address 127.0.0.1:54320)
0929 11:32:39.901513 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:39.901612 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 11:32:39.902819 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:39.902468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:38983 (local address 127.0.0.1:54332)
0929 11:32:39.902606 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:39.902682 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 11:32:39.903049 14587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.61:38983: Network error: Client connection negotiation failed: client connection to 127.23.153.61:38983: connect: Connection refused (error 111)
I20260929 11:32:39.994387 14587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } attempt: 3
I20260929 11:32:39.995973 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:39.995192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:38983 (local address 127.0.0.1:54342)
0929 11:32:39.995606 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:39.995771 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237}
I20260929 11:32:39.997547 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:39.997017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:38983 (local address 127.0.0.1:54356)
0929 11:32:39.997239 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:39.997343 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260929 11:32:39.997780 14587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.61:38983: Network error: Client connection negotiation failed: client connection to 127.23.153.61:38983: connect: Connection refused (error 111)
I20260929 11:32:40.141049 14587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } attempt: 4
I20260929 11:32:40.142541 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.141861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:38983 (local address 127.0.0.1:54358)
0929 11:32:40.142217 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.142343 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260929 11:32:40.144325 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.143768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:38983 (local address 127.0.0.1:54372)
0929 11:32:40.143986 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.144105 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.61:38983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 11:32:40.144572 14587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.61:38983: Network error: Client connection negotiation failed: client connection to 127.23.153.61:38983: connect: Connection refused (error 111)
W20260929 11:32:40.176617 14594 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 11:32:40.177014 14594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:40.177114 14594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:40.177198 14594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:40.187279 14594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:40.187565 14594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:40.187700 14594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:40.187775 14594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:40.187863 14594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:40.199389 14594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:39025,127.23.153.61:38983,127.23.153.60:40977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.61.keytab
--principal=kudu/127.23.153.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.61:38983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.153.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:40.200773 14594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:40.202143 14594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:40.214639 14602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:40.214823 14600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:40.214882 14599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:40.216183 14594 server_base.cc:1034] running on GCE node
I20260929 11:32:40.217015 14594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:40.218582 14594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:40.219849 14594 hybrid_clock.cc:648] HybridClock initialized: now 1790681560219772 us; error 51 us; skew 500 ppm
I20260929 11:32:40.220214 14594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:40 dist-test-slave-qk05 krb5kdc[14507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681560, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:40.223615 14594 init.cc:377] Logged in from keytab as kudu/127.23.153.61@KRBTEST.COM (short username kudu)
I20260929 11:32:40.225221 14594 webserver.cc:460] Webserver started at http://127.23.153.61:46177/ using document root <none> and password file <none>
I20260929 11:32:40.225970 14594 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:40.226081 14594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:40.226385 14594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:40.229596 14594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-1/data/instance:
uuid: "c81913f4d0194284a056743f4167a631"
format_stamp: "Formatted at 2026-09-29 11:32:40 on dist-test-slave-qk05"
server_key: "1a258621a021406fca44ac197b125736"
server_key_iv: "03ea188c1b2bca480732484c3bf7b3f6"
server_key_version: "encryptionkey@0"
I20260929 11:32:40.230459 14594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-1/wal/instance:
uuid: "c81913f4d0194284a056743f4167a631"
format_stamp: "Formatted at 2026-09-29 11:32:40 on dist-test-slave-qk05"
server_key: "1a258621a021406fca44ac197b125736"
server_key_iv: "03ea188c1b2bca480732484c3bf7b3f6"
server_key_version: "encryptionkey@0"
I20260929 11:32:40.236621 14594 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 11:32:40.241087 14609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:40.242643 14594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 11:32:40.242841 14594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-1/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-1/wal
uuid: "c81913f4d0194284a056743f4167a631"
format_stamp: "Formatted at 2026-09-29 11:32:40 on dist-test-slave-qk05"
server_key: "1a258621a021406fca44ac197b125736"
server_key_iv: "03ea188c1b2bca480732484c3bf7b3f6"
server_key_version: "encryptionkey@0"
I20260929 11:32:40.243187 14594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:40.296289 14594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:40.301014 14594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:40.301312 14594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:40.320976 14594 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.61:38983
I20260929 11:32:40.320991 14660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.61:38983 every 8 connection(s)
I20260929 11:32:40.322499 14594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-1/data/info.pb
I20260929 11:32:40.326460 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 14594
I20260929 11:32:40.326622 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-1/wal/instance
I20260929 11:32:40.326856 14661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:40.327107 24164 external_mini_cluster.cc:1428] Setting key 300fac0b8a0b6a45e06e863351387d1c
I20260929 11:32:40.334055 14661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:32:40 dist-test-slave-qk05 krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681560, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.61@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.153.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.153.60@KRBTEST.COM" created.
I20260929 11:32:40.357182 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.337013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46690 (local address 127.23.153.62:39025)
0929 11:32:40.337515 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:40.337520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:40.338751 (+ 1231us) server_negotiation.cc:407] Connection header received
0929 11:32:40.340845 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:40.340851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:40.340941 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:40.341040 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:40.342972 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.343755 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.345519 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.345961 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.346080 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:40.349994 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:40.350000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:40.350003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:40.350056 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:40.353379 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:40.354476 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:40.354481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:40.354483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:40.354706 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:40.355144 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:40.355149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:40.355153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:40.355506 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:40.355658 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:40.356530 (+ 872us) server_negotiation.cc:299] Negotiation successful
0929 11:32:40.356791 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":269,"threads_started":1}
I20260929 11:32:40.358250 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.337711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:39025 (local address 127.0.0.1:46690)
0929 11:32:40.338557 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.338602 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:40.340500 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:40.341428 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:40.341439 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:40.342047 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:40.342703 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:40.342723 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.343943 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.343947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:40.345336 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:40.345354 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.346226 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.346235 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:40.346471 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:40.347179 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:40.347208 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:40.349697 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:40.353591 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:40.353600 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:40.353618 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:40.354314 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:40.354807 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:40.354810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:40.354813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:40.355042 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:40.355662 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:40.355672 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:40.356095 (+ 423us) client_negotiation.cc:769] Sending connection context
0929 11:32:40.356383 (+ 288us) client_negotiation.cc:240] Negotiation successful
0929 11:32:40.357001 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":345,"threads_started":1}
I20260929 11:32:40.360790 14661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:32:40 dist-test-slave-qk05 krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681560, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.61@KRBTEST.COM for kudu/127.23.153.61@KRBTEST.COM
I20260929 11:32:40.377422 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.361961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:38983 (local address 127.0.0.1:54380)
0929 11:32:40.362306 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.362329 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:40.362457 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:40.363543 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:40.363550 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:40.364336 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:40.364538 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:40.364553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.366221 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.366225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:40.367158 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:40.367172 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.368047 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.368061 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:40.368250 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:40.368835 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:40.368855 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:40.371222 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:40.374611 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:40.374617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:40.374621 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:40.375183 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:40.375583 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:40.375587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:40.375590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:40.375762 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:40.376392 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:40.376397 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:40.376553 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 11:32:40.376698 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 11:32:40.377022 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 11:32:40.377457 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.362220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54380 (local address 127.23.153.61:38983)
0929 11:32:40.362905 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:40.362912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:40.362942 (+ 30us) server_negotiation.cc:407] Connection header received
0929 11:32:40.362993 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:40.363000 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:40.363249 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:40.363380 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:40.364778 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.365970 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.367306 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.367895 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.368015 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:40.371477 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:40.371493 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:40.371510 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:40.371545 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:40.374417 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:40.375323 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:40.375331 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:40.375334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:40.375472 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:40.375872 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:40.375876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:40.375877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:40.376259 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:40.376430 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:40.376825 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 11:32:40.377141 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":527,"thread_start_us":392,"threads_started":1}
I20260929 11:32:40.380110 14661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:32:40.381387 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.380779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53168)
0929 11:32:40.380988 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.381143 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 11:32:40.381692 14611 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
I20260929 11:32:40.383451 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.383087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53184)
0929 11:32:40.383236 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.383328 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 11:32:40.383670 14661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:40977: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.153.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.60.keytab.
Entry for principal kudu/127.23.153.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.60.keytab.
I20260929 11:32:40.390105 24164 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
/tmp/dist-test-taskBSjPgi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.60:40977
--webserver_interface=127.23.153.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.60.keytab
--principal=kudu/127.23.153.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.153.20:41973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.153.62:39025,127.23.153.61:38983,127.23.153.60:40977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 11:32:40.433015 14661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } attempt: 1
I20260929 11:32:40.434393 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.433791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53192)
0929 11:32:40.434033 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.434184 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 11:32:40.435799 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.435347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53206)
0929 11:32:40.435531 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.435640 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 11:32:40.436102 14661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:40977: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
I20260929 11:32:40.443861 14587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } attempt: 5
Sep 29 11:32:40 dist-test-slave-qk05 krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681559, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for kudu/127.23.153.61@KRBTEST.COM
I20260929 11:32:40.459702 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.445044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54382 (local address 127.23.153.61:38983)
0929 11:32:40.445253 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:40.445258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:40.445274 (+ 16us) server_negotiation.cc:407] Connection header received
0929 11:32:40.445343 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:40.445347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:40.445409 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:40.445493 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:40.446334 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.447130 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.448360 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.448831 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.448950 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:40.452746 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:40.452752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:40.452755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:40.452797 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:40.456766 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:40.457597 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:40.457604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:40.457608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:40.457752 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:40.458160 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:40.458165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:40.458169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:40.458521 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:40.458660 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:40.459122 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 11:32:40.459363 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 11:32:40.459707 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.444752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:38983 (local address 127.0.0.1:54382)
0929 11:32:40.444978 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.444995 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:40.445111 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:40.445550 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:40.445563 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:40.445987 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:40.446172 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:40.446182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.447271 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.447277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:40.448162 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:40.448180 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.449011 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.449018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:40.449335 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:40.449981 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:40.450001 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:40.452457 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:40.456995 (+ 4538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:40.457002 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:40.457005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:40.457455 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:40.457872 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:40.457877 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:40.457880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:40.458047 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:40.458651 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:40.458656 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:40.458817 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 11:32:40.458962 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 11:32:40.459360 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 11:32:40.461162 14587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } has no permanent_uuid. Determining permanent_uuid...
I20260929 11:32:40.463076 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.462165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53210)
0929 11:32:40.462528 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.462698 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 11:32:40.464273 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.463777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53224)
0929 11:32:40.463981 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.464111 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 11:32:40.464545 14587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:40977: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
I20260929 11:32:40.504387 14661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } attempt: 2
I20260929 11:32:40.506027 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.505317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53230)
0929 11:32:40.505644 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.505789 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260929 11:32:40.507449 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.506967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53234)
0929 11:32:40.507179 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.507279 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260929 11:32:40.507736 14661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:40977: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
I20260929 11:32:40.515798 14587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } attempt: 1
I20260929 11:32:40.517448 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.516753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53240)
0929 11:32:40.517063 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.517218 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 11:32:40.518745 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.518305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53254)
0929 11:32:40.518488 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.518594 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 11:32:40.519060 14587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:40977: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
I20260929 11:32:40.587344 14587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } attempt: 2
I20260929 11:32:40.588745 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.588151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53256)
0929 11:32:40.588402 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.588531 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 11:32:40.589905 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.589479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53270)
0929 11:32:40.589653 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.589751 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 11:32:40.590153 14587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:40977: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
I20260929 11:32:40.599006 14661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } attempt: 3
I20260929 11:32:40.600522 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.599887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53278)
0929 11:32:40.600139 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.600268 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 11:32:40.601989 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.601366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53284)
0929 11:32:40.601663 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.601789 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
W20260929 11:32:40.602303 14661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:40977: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
I20260929 11:32:40.696383 14587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } attempt: 3
I20260929 11:32:40.697647 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.697086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53300)
0929 11:32:40.697323 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.697434 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 11:32:40.698812 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.698394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53302)
0929 11:32:40.698577 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.698677 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 11:32:40.699038 14587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:40977: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
I20260929 11:32:40.745548 14661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } attempt: 4
I20260929 11:32:40.746879 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.746297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53314)
0929 11:32:40.746526 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.746634 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 11:32:40.748158 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.747769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53320)
0929 11:32:40.747926 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.748015 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 11:32:40.748377 14661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:40977: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
W20260929 11:32:40.769925 14669 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 11:32:40.770277 14669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 11:32:40.770372 14669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 11:32:40.770480 14669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 11:32:40.778157 14669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 11:32:40.778266 14669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 11:32:40.778326 14669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 11:32:40.778395 14669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 11:32:40.778476 14669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 11:32:40.789016 14669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.153.20:41973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.153.62:39025,127.23.153.61:38983,127.23.153.60:40977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/krb5kdc/kudu_127.23.153.60.keytab
--principal=kudu/127.23.153.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.153.60:40977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.153.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 10:22:15 UTC on ccycloud.quasar-narmil.root.comops.site
build id 4679903
ASAN enabled
I20260929 11:32:40.790385 14669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 11:32:40.791772 14669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 11:32:40.803782 14674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:40.803717 14675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 11:32:40.803737 14677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 11:32:40.805040 14669 server_base.cc:1034] running on GCE node
I20260929 11:32:40.805895 14669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 11:32:40.807534 14669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 11:32:40.808827 14669 hybrid_clock.cc:648] HybridClock initialized: now 1790681560808748 us; error 43 us; skew 500 ppm
I20260929 11:32:40.809245 14669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 11:32:40 dist-test-slave-qk05 krb5kdc[14507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681560, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 11:32:40.812965 14669 init.cc:377] Logged in from keytab as kudu/127.23.153.60@KRBTEST.COM (short username kudu)
I20260929 11:32:40.814694 14669 webserver.cc:460] Webserver started at http://127.23.153.60:42877/ using document root <none> and password file <none>
I20260929 11:32:40.815506 14669 fs_manager.cc:359] Metadata directory not provided
I20260929 11:32:40.815614 14669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 11:32:40.815907 14669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 11:32:40.819232 14669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-2/data/instance:
uuid: "2c2abc210c01442daad9c43a85f25297"
format_stamp: "Formatted at 2026-09-29 11:32:40 on dist-test-slave-qk05"
server_key: "79555e7e739cbdb4a60e4830705bc539"
server_key_iv: "1e9c09c8184d7e49ad567ece4c72a097"
server_key_version: "encryptionkey@0"
I20260929 11:32:40.820102 14669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-2/wal/instance:
uuid: "2c2abc210c01442daad9c43a85f25297"
format_stamp: "Formatted at 2026-09-29 11:32:40 on dist-test-slave-qk05"
server_key: "79555e7e739cbdb4a60e4830705bc539"
server_key_iv: "1e9c09c8184d7e49ad567ece4c72a097"
server_key_version: "encryptionkey@0"
I20260929 11:32:40.826086 14669 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 11:32:40.830277 14684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 11:32:40.832710 14669 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 11:32:40.832976 14669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-2/data,/tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-2/wal
uuid: "2c2abc210c01442daad9c43a85f25297"
format_stamp: "Formatted at 2026-09-29 11:32:40 on dist-test-slave-qk05"
server_key: "79555e7e739cbdb4a60e4830705bc539"
server_key_iv: "1e9c09c8184d7e49ad567ece4c72a097"
server_key_version: "encryptionkey@0"
I20260929 11:32:40.833263 14669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 11:32:40.856344 14669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 11:32:40.861650 14669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 11:32:40.861938 14669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 11:32:40.876298 14587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } attempt: 4
I20260929 11:32:40.877866 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.877203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53334)
0929 11:32:40.877545 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.877659 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260929 11:32:40.879179 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.878614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53344)
0929 11:32:40.878820 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.878930 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 11:32:40.879551 14587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.153.60:40977: Network error: Client connection negotiation failed: client connection to 127.23.153.60:40977: connect: Connection refused (error 111)
I20260929 11:32:40.882967 14669 rpc_server.cc:307] RPC server started. Bound to: 127.23.153.60:40977
I20260929 11:32:40.882943 14735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.153.60:40977 every 8 connection(s)
I20260929 11:32:40.884485 14669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-2/data/info.pb
I20260929 11:32:40.888563 14736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 11:32:40.890472 24164 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu as pid 14669
I20260929 11:32:40.890687 24164 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBSjPgi/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790681181154781-24164-0/minicluster-data/master-2/wal/instance
I20260929 11:32:40.891276 24164 external_mini_cluster.cc:1428] Setting key 537f745459b6979e8c24621a5a71ef13
I20260929 11:32:40.899720 14736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:32:40 dist-test-slave-qk05 krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.60@KRBTEST.COM
Sep 29 11:32:40 dist-test-slave-qk05 krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681560, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.60@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:32:40.915419 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.893382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53360 (local address 127.23.153.60:40977)
0929 11:32:40.894530 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:40.894550 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:40.894634 (+ 84us) server_negotiation.cc:407] Connection header received
0929 11:32:40.896532 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:40.896562 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:40.897070 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:40.897912 (+ 842us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:40.899665 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.901150 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.902976 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.903990 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.904173 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:40.908005 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:40.908014 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:40.908028 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:40.908077 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:40.911142 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:40.911904 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:40.911912 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:40.911919 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:40.912090 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:40.912452 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:40.912456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:40.912458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:40.913029 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:40.913333 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:40.913817 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 11:32:40.914291 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":400,"threads_started":1}
I20260929 11:32:40.917682 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.902375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.62:39025 (local address 127.0.0.1:46694)
0929 11:32:40.903209 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.903259 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:40.903406 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:40.904124 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:40.904132 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:40.904766 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:40.905027 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:40.905037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.906074 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.906077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:40.907184 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:40.907205 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.908005 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.908014 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:40.908173 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:40.908812 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:40.908857 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:40.911238 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:40.914961 (+ 3723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:40.914974 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:40.914980 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:40.915488 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:40.916031 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:40.916035 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:40.916037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:40.916197 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:40.916748 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:40.916754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:40.916940 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 11:32:40.917094 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 11:32:40.917373 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":371,"threads_started":1}
I20260929 11:32:40.918041 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.902508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46694 (local address 127.23.153.62:39025)
0929 11:32:40.903796 (+ 1288us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:40.903800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:40.903816 (+ 16us) server_negotiation.cc:407] Connection header received
0929 11:32:40.903873 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:40.903876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:40.903937 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:40.904031 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:40.905179 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.905928 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.907341 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.907745 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.907859 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:40.911461 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:40.911466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:40.911469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:40.911508 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:40.914807 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:40.915713 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:40.915719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:40.915722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:40.915911 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:40.916312 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:40.916318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:40.916321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:40.916613 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:40.916784 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:40.917246 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 11:32:40.917505 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1185,"thread_start_us":1045,"threads_started":1}
I20260929 11:32:40.919659 14736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:32:40 dist-test-slave-qk05 krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681560, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.60@KRBTEST.COM for kudu/127.23.153.61@KRBTEST.COM
Sep 29 11:32:40 dist-test-slave-qk05 krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.62@KRBTEST.COM
I20260929 11:32:40.934701 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.920538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.61:38983 (local address 127.0.0.1:54390)
0929 11:32:40.920732 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.920753 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:40.920859 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:40.921217 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:40.921221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:40.921595 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:40.921746 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:40.921755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.922722 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.922727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:40.923821 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:40.923838 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.924616 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.924622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:40.924825 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:40.925346 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:40.925369 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:40.928058 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:40.931887 (+ 3829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:40.931892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:40.931895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:40.932334 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:40.932893 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:40.932898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:40.932901 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:40.933123 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:40.933790 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:40.933794 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:40.933976 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 11:32:40.934144 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 11:32:40.934431 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 11:32:40.935645 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.920654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.23.153.61:38983)
0929 11:32:40.920858 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:40.920869 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:40.920887 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:32:40.920949 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:40.920954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:40.921033 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:40.921151 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:40.921920 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.922583 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.923997 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.924466 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.924587 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:40.928307 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:40.928313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:40.928316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:40.928356 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:40.931663 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:40.932498 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:40.932505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:40.932508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:40.932714 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:40.933284 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:40.933288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:40.933290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:40.933660 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:40.933820 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:40.934450 (+ 630us) server_negotiation.cc:299] Negotiation successful
0929 11:32:40.935252 (+ 802us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 11:32:40.936405 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.918701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.23.153.62:39025)
0929 11:32:40.918843 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:40.918848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:40.918932 (+ 84us) server_negotiation.cc:407] Connection header received
0929 11:32:40.919213 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:40.919219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:40.919312 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:40.919425 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:40.920959 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.921774 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.923242 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.923709 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.923817 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:40.928928 (+ 5111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:40.928934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:40.928937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:40.928977 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:40.932722 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:40.933626 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:40.933632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:40.933635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:40.933801 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:40.934362 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:40.934371 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:40.934375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:40.934936 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:40.935497 (+ 561us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:40.935715 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 11:32:40.936098 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 11:32:40.936949 14736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 11:32:40 dist-test-slave-qk05 krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681560, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.60@KRBTEST.COM for kudu/127.23.153.60@KRBTEST.COM
Sep 29 11:32:40 dist-test-slave-qk05 krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.153.61@KRBTEST.COM
I20260929 11:32:40.954322 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.937734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.23.153.60:40977)
0929 11:32:40.938268 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:40.938274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:40.938446 (+ 172us) server_negotiation.cc:407] Connection header received
0929 11:32:40.938602 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:40.938607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:40.938674 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:40.938754 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:40.940002 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.940964 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.943277 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.943885 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.944042 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:40.948191 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:40.948198 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:40.948204 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:40.948243 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:40.951288 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:40.952108 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:40.952112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:40.952115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:40.952248 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:40.952661 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:40.952665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:40.952667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:40.952996 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:40.953151 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:40.953559 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 11:32:40.953980 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":175,"server-negotiator.queue_time_us":415}
I20260929 11:32:40.954322 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.937678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53374)
0929 11:32:40.938360 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:40.938377 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:40.938483 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:40.938947 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:40.938951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:40.939635 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:40.939848 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:40.939863 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.941234 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.941241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:40.943005 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:40.943061 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.944079 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.944083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:40.944334 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:40.945191 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:40.945217 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:40.947955 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:40.951499 (+ 3544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:40.951506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:40.951511 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:40.951960 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:40.952379 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:40.952384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:40.952388 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:40.952540 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:40.953123 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:40.953129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:40.953284 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 11:32:40.953434 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 11:32:40.953812 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":1}
I20260929 11:32:40.957314 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:40.938066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54392 (local address 127.23.153.61:38983)
0929 11:32:40.938220 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:40.938225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:40.938243 (+ 18us) server_negotiation.cc:407] Connection header received
0929 11:32:40.938405 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:40.938410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:40.938537 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:40.938646 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:40.940405 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.941319 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.943474 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:40.944308 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:40.944432 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:40.949667 (+ 5235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:40.949676 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:40.949680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:40.949736 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:40.953320 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:40.954253 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:40.954260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:40.954262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:40.954418 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:40.955308 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:40.955317 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:40.955321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:40.955790 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:40.956212 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:40.956464 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 11:32:40.956837 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 11:32:40.960420 14736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297: Bootstrap starting.
I20260929 11:32:40.962982 24164 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 11:32:40.964371 14736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:40.965687 14736 log.cc:824] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:40.965740 14562 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7d75014d97084589a5e483793a728b61"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46700
I20260929 11:32:40.968794 14736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297: No bootstrap required, opened a new log
I20260929 11:32:40.978454 14736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d75014d97084589a5e483793a728b61" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } } peers { permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } } peers { permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } }
I20260929 11:32:40.978854 14736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2abc210c01442daad9c43a85f25297, State: Initialized, Role: FOLLOWER
I20260929 11:32:40.979386 14736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d75014d97084589a5e483793a728b61" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } } peers { permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } } peers { permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } }
I20260929 11:32:40.980386 14743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d75014d97084589a5e483793a728b61" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } } peers { permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } } peers { permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } } }
I20260929 11:32:40.980769 14743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:32:40.981751 14736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 11:32:40.992283 14754 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:32:40.992473 14754 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 11:32:40.989646 14669 master_runner.cc:186] Error getting master registration for 127.23.153.62:39025: OK, instance_id { permanent_uuid: "7d75014d97084589a5e483793a728b61" instance_seqno: 1790681559721688 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 11:32:41.003434 14669 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 11:32:41.047636 14661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } attempt: 5
Sep 29 11:32:41 dist-test-slave-qk05 krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681560, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.61@KRBTEST.COM for kudu/127.23.153.60@KRBTEST.COM
I20260929 11:32:41.070158 14665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:41.048342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53390)
0929 11:32:41.048547 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:41.048567 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:41.048690 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:41.049078 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:41.049083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:41.049624 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:41.049841 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:41.049855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:41.051482 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:41.051488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:41.052915 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:41.052935 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:41.054075 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:41.054080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:41.054267 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:41.059948 (+ 5681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:41.059979 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:41.062356 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:41.066881 (+ 4525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:41.066889 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:41.066894 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:41.067576 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:41.068084 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:41.068090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:41.068093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:41.068278 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:41.069048 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:41.069053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:41.069238 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 11:32:41.069397 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 11:32:41.069747 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 11:32:41.070271 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:41.048464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53390 (local address 127.23.153.60:40977)
0929 11:32:41.048678 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:41.048684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:41.048710 (+ 26us) server_negotiation.cc:407] Connection header received
0929 11:32:41.048824 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:41.048830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:41.048919 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:41.049051 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:41.050035 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:41.051310 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:41.053134 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:41.053898 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:41.054066 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:41.062636 (+ 8570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:41.062644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:41.062649 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:41.062699 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:41.066637 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:41.067751 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:41.067758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:41.067762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:41.067947 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:41.068388 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:41.068394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:41.068397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:41.068927 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:41.069110 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:41.069531 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 11:32:41.069791 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 11:32:41.076455 14661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631: Bootstrap starting.
I20260929 11:32:41.080757 14661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:41.082190 14661 log.cc:824] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:41.085716 14661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631: No bootstrap required, opened a new log
I20260929 11:32:41.097039 14661 raft_consensus.cc:348] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d75014d97084589a5e483793a728b61" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } } peers { permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } } peers { permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } }
I20260929 11:32:41.097520 14661 raft_consensus.cc:729] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c81913f4d0194284a056743f4167a631, State: Initialized, Role: FOLLOWER
I20260929 11:32:41.098125 14661 consensus_queue.cc:260] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d75014d97084589a5e483793a728b61" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } } peers { permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } } peers { permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } }
I20260929 11:32:41.099469 14757 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d75014d97084589a5e483793a728b61" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } } peers { permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } } peers { permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } } }
I20260929 11:32:41.099928 14757 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:32:41.101239 14661 sys_catalog.cc:564] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 11:32:41.112350 14768 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 11:32:41.112612 14768 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 11:32:41.114353 14690 tablet.cc:2369] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:32:41.109215 14594 master_runner.cc:186] Error getting master registration for 127.23.153.62:39025: OK, instance_id { permanent_uuid: "7d75014d97084589a5e483793a728b61" instance_seqno: 1790681559721688 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 11:32:41.123865 14594 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 11:32:41.156790 14587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } attempt: 5
Sep 29 11:32:41 dist-test-slave-qk05 krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790681559, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.153.62@KRBTEST.COM for kudu/127.23.153.60@KRBTEST.COM
I20260929 11:32:41.173137 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:41.157720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.23.153.60:40977)
0929 11:32:41.157952 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:41.157958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:41.158218 (+ 260us) server_negotiation.cc:407] Connection header received
0929 11:32:41.158287 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:41.158292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:41.158372 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:41.158475 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:41.159520 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:41.160303 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:41.162218 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:41.162715 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:41.162847 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:41.166745 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:41.166751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:41.166755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:41.166794 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:41.170001 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:41.170944 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:41.170950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:41.170952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:41.171209 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:41.171672 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:41.171676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:41.171678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:41.171960 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:41.172108 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:41.172543 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 11:32:41.172772 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 11:32:41.173170 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:41.157573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.153.60:40977 (local address 127.0.0.1:53406)
0929 11:32:41.157943 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 11:32:41.157965 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 11:32:41.158112 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 11:32:41.158511 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 11:32:41.158516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 11:32:41.158950 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 11:32:41.159325 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:41.159343 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:41.160468 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:41.160474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:41.162024 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 11:32:41.162045 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 11:32:41.162903 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 11:32:41.162909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 11:32:41.163166 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:41.163876 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 11:32:41.163896 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 11:32:41.166487 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 11:32:41.170228 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:41.170235 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:41.170239 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:41.170811 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:41.171341 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 11:32:41.171347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 11:32:41.171351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 11:32:41.171556 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 11:32:41.172088 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 11:32:41.172093 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 11:32:41.172276 (+ 183us) client_negotiation.cc:769] Sending connection context
0929 11:32:41.172421 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 11:32:41.172747 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252}
I20260929 11:32:41.178311 14587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61: Bootstrap starting.
I20260929 11:32:41.182880 14587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61: Neither blocks nor log segments found. Creating new log.
I20260929 11:32:41.184397 14587 log.cc:824] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61: Log is configured to *not* fsync() on all Append() calls
I20260929 11:32:41.188548 14587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61: No bootstrap required, opened a new log
I20260929 11:32:41.197216 14587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d75014d97084589a5e483793a728b61" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } } peers { permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } } peers { permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } }
I20260929 11:32:41.197710 14587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d75014d97084589a5e483793a728b61, State: Initialized, Role: FOLLOWER
I20260929 11:32:41.198277 14587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d75014d97084589a5e483793a728b61" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } } peers { permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } } peers { permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } }
I20260929 11:32:41.199740 14771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d75014d97084589a5e483793a728b61" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } } peers { permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } } peers { permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } } }
I20260929 11:32:41.200474 14771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:32:41.201146 14587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 11:32:41.201210 14562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 11:32:41.201414 14562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:41.205220 14562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d75014d97084589a5e483793a728b61" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } } peers { permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } } peers { permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } }
I20260929 11:32:41.208215 14636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d75014d97084589a5e483793a728b61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c81913f4d0194284a056743f4167a631"
I20260929 11:32:41.208402 14711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d75014d97084589a5e483793a728b61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2c2abc210c01442daad9c43a85f25297"
I20260929 11:32:41.208828 14636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:41.208868 14711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297 [term 0 FOLLOWER]: Advancing to term 1
I20260929 11:32:41.212474 14711 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d75014d97084589a5e483793a728b61 in term 1.
I20260929 11:32:41.212474 14636 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d75014d97084589a5e483793a728b61 in term 1.
W20260929 11:32:41.213238 14541 tablet.cc:2369] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:32:41.213896 14537 leader_election.cc:302] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c2abc210c01442daad9c43a85f25297, 7d75014d97084589a5e483793a728b61; no voters:
I20260929 11:32:41.214217 14562 leader_election.cc:288] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [CANDIDATE]: Term 1 election: Requested vote from peers c81913f4d0194284a056743f4167a631 (127.23.153.61:38983), 2c2abc210c01442daad9c43a85f25297 (127.23.153.60:40977)
I20260929 11:32:41.214933 14771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 11:32:41.228149 14771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [term 1 LEADER]: Becoming Leader. State: Replica: 7d75014d97084589a5e483793a728b61, State: Running, Role: LEADER
I20260929 11:32:41.229097 14771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d75014d97084589a5e483793a728b61" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } } peers { permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } } peers { permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } }
I20260929 11:32:41.239060 14773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d75014d97084589a5e483793a728b61. Latest consensus state: current_term: 1 leader_uuid: "7d75014d97084589a5e483793a728b61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d75014d97084589a5e483793a728b61" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } } peers { permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } } peers { permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } } }
I20260929 11:32:41.239672 14773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:41.247045 14782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 11:32:41.276782 14782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260929 11:32:41.304346 14615 tablet.cc:2369] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 11:32:41.306294 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:41.246356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.23.153.61:38983)
0929 11:32:41.246550 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:41.246557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:41.259493 (+ 12936us) server_negotiation.cc:407] Connection header received
0929 11:32:41.264074 (+ 4581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:41.264081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:41.264191 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:41.264281 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:41.270763 (+ 6482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:41.272008 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:41.277773 (+ 5765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:41.278865 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:41.278995 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:41.289529 (+ 10534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:41.289546 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:41.289552 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:41.289618 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:41.294428 (+ 4810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:41.300803 (+ 6375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:41.300811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:41.300815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:41.301037 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:41.302391 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:41.302398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:41.302402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:41.302896 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:41.303197 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:41.305220 (+ 2023us) server_negotiation.cc:299] Negotiation successful
0929 11:32:41.305530 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 11:32:41.308142 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:41.255592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46704 (local address 127.23.153.62:39025)
0929 11:32:41.255809 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:41.255816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:41.263697 (+ 7881us) server_negotiation.cc:407] Connection header received
0929 11:32:41.266726 (+ 3029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:41.266738 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:41.266838 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:41.266913 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:41.274747 (+ 7834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:41.275888 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:41.279837 (+ 3949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:41.280925 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:41.281069 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:41.295605 (+ 14536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:41.295612 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:41.295617 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:41.295675 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:41.299689 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:41.301642 (+ 1953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:41.301652 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:41.301656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:41.301833 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:41.303746 (+ 1913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:41.303752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:41.303755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:41.304185 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:41.304563 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:41.307481 (+ 2918us) server_negotiation.cc:299] Negotiation successful
0929 11:32:41.307759 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 11:32:41.316365 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:41.260374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53408 (local address 127.23.153.60:40977)
0929 11:32:41.260614 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:41.260620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:41.272626 (+ 12006us) server_negotiation.cc:407] Connection header received
0929 11:32:41.273800 (+ 1174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:41.273807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:41.273937 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:41.274017 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:41.282149 (+ 8132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:41.283269 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:41.292863 (+ 9594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:41.293687 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:41.293863 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:41.304916 (+ 11053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 11:32:41.304923 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 11:32:41.304928 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 11:32:41.304973 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 11:32:41.308835 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:41.313466 (+ 4631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:41.313472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:41.313475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:41.313680 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 11:32:41.314181 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 11:32:41.314187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 11:32:41.314190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 11:32:41.314679 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 11:32:41.314959 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:41.315624 (+ 665us) server_negotiation.cc:299] Negotiation successful
0929 11:32:41.315943 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 11:32:41.319948 14711 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297 [term 1 FOLLOWER]: Refusing update from remote peer 7d75014d97084589a5e483793a728b61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:32:41.319936 14636 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631 [term 1 FOLLOWER]: Refusing update from remote peer 7d75014d97084589a5e483793a728b61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 11:32:41.321287 14773 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:32:41.321623 14771 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 11:32:41.344791 14757 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d75014d97084589a5e483793a728b61. Latest consensus state: current_term: 1 leader_uuid: "7d75014d97084589a5e483793a728b61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d75014d97084589a5e483793a728b61" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } } peers { permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } } peers { permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } } }
I20260929 11:32:41.347275 14757 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:32:41.355348 14795 mvcc.cc:205] Tried to move back new op lower bound from 7334631675132469248 to 7334631674804576256. Current Snapshot: MvccSnapshot[applied={T|T < 7334631675132469248}]
I20260929 11:32:41.360601 14757 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d75014d97084589a5e483793a728b61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d75014d97084589a5e483793a728b61" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } } peers { permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } } peers { permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } } }
I20260929 11:32:41.360954 14757 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:32:41.361987 14792 mvcc.cc:205] Tried to move back new op lower bound from 7334631675132469248 to 7334631674804576256. Current Snapshot: MvccSnapshot[applied={T|T < 7334631675132469248}]
I20260929 11:32:41.371924 14771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d75014d97084589a5e483793a728b61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d75014d97084589a5e483793a728b61" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } } peers { permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } } peers { permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } } }
I20260929 11:32:41.372364 14771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:41.373045 14794 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 11:32:41.377965 14782 catalog_manager.cc:1347] Generated new cluster ID: 26f4ecc6c6074e13985abee30f95215c
I20260929 11:32:41.380031 14782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 11:32:41.383204 14743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d75014d97084589a5e483793a728b61. Latest consensus state: current_term: 1 leader_uuid: "7d75014d97084589a5e483793a728b61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d75014d97084589a5e483793a728b61" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } } peers { permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } } peers { permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } } }
I20260929 11:32:41.383541 14743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:32:41.384558 14773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d75014d97084589a5e483793a728b61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d75014d97084589a5e483793a728b61" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } } peers { permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } } peers { permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } } }
I20260929 11:32:41.384974 14773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61 [sys.catalog]: This master's current role is: LEADER
I20260929 11:32:41.396257 14743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d75014d97084589a5e483793a728b61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d75014d97084589a5e483793a728b61" member_type: VOTER last_known_addr { host: "127.23.153.62" port: 39025 } } peers { permanent_uuid: "c81913f4d0194284a056743f4167a631" member_type: VOTER last_known_addr { host: "127.23.153.61" port: 38983 } } peers { permanent_uuid: "2c2abc210c01442daad9c43a85f25297" member_type: VOTER last_known_addr { host: "127.23.153.60" port: 40977 } } }
I20260929 11:32:41.396569 14743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 11:32:41.440384 14782 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 11:32:41.442756 14782 catalog_manager.cc:1504] Loading token signing keys...
I20260929 11:32:41.464133 14782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d75014d97084589a5e483793a728b61: Generated new TSK 0
I20260929 11:32:41.465350 14782 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 11:32:41.602322 14666 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:41.586877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.23.153.61:38983)
0929 11:32:41.587136 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:41.587143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:41.600068 (+ 12925us) server_negotiation.cc:407] Connection header received
0929 11:32:41.600402 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:41.600408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:41.600502 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:41.600581 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:41.602057 (+ 1476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54402: BlockingRecv error: recv got EOF from 127.0.0.1:54402 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 11:32:41.602315 14741 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:41.593621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46720 (local address 127.23.153.62:39025)
0929 11:32:41.593893 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:41.593899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:41.600393 (+ 6494us) server_negotiation.cc:407] Connection header received
0929 11:32:41.600631 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:41.600638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:41.600720 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:41.600786 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:41.602057 (+ 1271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46720: BlockingRecv error: recv got EOF from 127.0.0.1:46720 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260929 11:32:41.605023 14739 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:41.599939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.23.153.60:40977)
0929 11:32:41.600122 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:41.600127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:41.603141 (+ 3014us) server_negotiation.cc:407] Connection header received
0929 11:32:41.603411 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:41.603416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:41.603507 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:41.603584 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:41.604753 (+ 1169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53424: BlockingRecv error: recv got EOF from 127.0.0.1:53424 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 11:32:41.621657 14666 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:41.613985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.23.153.61:38983)
0929 11:32:41.614193 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:41.614197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:41.619799 (+ 5602us) server_negotiation.cc:407] Connection header received
0929 11:32:41.620075 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:41.620082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:41.620239 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:41.620418 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:41.621316 (+ 898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54404: BlockingRecv error: recv got EOF from 127.0.0.1:54404 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 11:32:41.635818 14666 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:41.633533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.23.153.61:38983)
0929 11:32:41.633746 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:41.633752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:41.633972 (+ 220us) server_negotiation.cc:407] Connection header received
0929 11:32:41.634336 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:41.634342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:41.634447 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:41.634566 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:41.635554 (+ 988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54416: BlockingRecv error: recv got EOF from 127.0.0.1:54416 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 11:32:41.648074 14666 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:41.645521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.23.153.61:38983)
0929 11:32:41.645729 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:41.645735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:41.646280 (+ 545us) server_negotiation.cc:407] Connection header received
0929 11:32:41.646491 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:41.646497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:41.646618 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:41.646724 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:41.647810 (+ 1086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54420: BlockingRecv error: recv got EOF from 127.0.0.1:54420 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 11:32:41.659585 14666 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:41.657212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.23.153.61:38983)
0929 11:32:41.657430 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:41.657437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:41.658015 (+ 578us) server_negotiation.cc:407] Connection header received
0929 11:32:41.658207 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:41.658213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:41.658317 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:41.658441 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:41.659306 (+ 865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54428: BlockingRecv error: recv got EOF from 127.0.0.1:54428 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 11:32:41.675799 14666 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:41.673090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.23.153.61:38983)
0929 11:32:41.673285 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:41.673291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:41.673893 (+ 602us) server_negotiation.cc:407] Connection header received
0929 11:32:41.674256 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:41.674263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:41.674402 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:41.674578 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:41.675501 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54442: BlockingRecv error: recv got EOF from 127.0.0.1:54442 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 11:32:41.697767 14666 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:41.695434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.23.153.61:38983)
0929 11:32:41.695636 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:41.695643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:41.695913 (+ 270us) server_negotiation.cc:407] Connection header received
0929 11:32:41.696252 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:41.696262 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:41.696489 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:41.696631 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:41.697525 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54444: BlockingRecv error: recv got EOF from 127.0.0.1:54444 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 11:32:41.736270 14666 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:41.733876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54456 (local address 127.23.153.61:38983)
0929 11:32:41.734115 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:41.734122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:41.734571 (+ 449us) server_negotiation.cc:407] Connection header received
0929 11:32:41.734778 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:41.734782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:41.734878 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:41.734971 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:41.735881 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54456: BlockingRecv error: recv got EOF from 127.0.0.1:54456 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 11:32:41.806198 14666 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:41.804106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54464 (local address 127.23.153.61:38983)
0929 11:32:41.804306 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:41.804312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:41.804750 (+ 438us) server_negotiation.cc:407] Connection header received
0929 11:32:41.804957 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:41.804961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:41.805051 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:41.805171 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:41.805972 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54464: BlockingRecv error: recv got EOF from 127.0.0.1:54464 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 11:32:41.941573 14666 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 11:32:41.939681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.23.153.61:38983)
0929 11:32:41.939916 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:41.939922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:41.940213 (+ 291us) server_negotiation.cc:407] Connection header received
0929 11:32:41.940407 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:41.940411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:41.940491 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:41.940575 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 11:32:41.941322 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54474: BlockingRecv error: recv got EOF from 127.0.0.1:54474 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 11:32:41.994539 14754 catalog_manager.cc:1259] Loaded cluster ID: 26f4ecc6c6074e13985abee30f95215c
I20260929 11:32:41.994666 14754 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297: loading cluster ID for follower catalog manager: success
I20260929 11:32:41.997287 14754 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297: acquiring CA information for follower catalog manager: success
I20260929 11:32:41.999459 14754 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2c2abc210c01442daad9c43a85f25297: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 11:32:42.114705 14768 catalog_manager.cc:1259] Loaded cluster ID: 26f4ecc6c6074e13985abee30f95215c
I20260929 11:32:42.114827 14768 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631: loading cluster ID for follower catalog manager: success
I20260929 11:32:42.117518 14768 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631: acquiring CA information for follower catalog manager: success
I20260929 11:32:42.119627 14768 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c81913f4d0194284a056743f4167a631: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 11:32:42.207593 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:42.201298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54478 (local address 127.23.153.61:38983)
0929 11:32:42.201494 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:42.201501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:42.201879 (+ 378us) server_negotiation.cc:407] Connection header received
0929 11:32:42.202175 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:42.202182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:42.202394 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:42.202498 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:32:42.202929 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:42.203795 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:42.205259 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:42.205700 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:42.205821 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:42.206340 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:32:42.206912 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:32:42.207093 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:42.207215 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 11:32:42.207309 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 11:32:42.223601 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:42.215492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.23.153.60:40977)
0929 11:32:42.215963 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:42.215968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:42.216979 (+ 1011us) server_negotiation.cc:407] Connection header received
0929 11:32:42.217371 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:42.217375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:42.217547 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:42.217737 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:32:42.218303 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:42.219114 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:42.220997 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:42.221578 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:42.221702 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:42.222362 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:32:42.222920 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:32:42.223118 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:42.223247 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 11:32:42.223337 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":261,"threads_started":1}
I20260929 11:32:42.238602 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:42.230305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46736 (local address 127.23.153.62:39025)
0929 11:32:42.230773 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:42.230779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:42.231326 (+ 547us) server_negotiation.cc:407] Connection header received
0929 11:32:42.231653 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:42.231659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:42.231908 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:42.232171 (+ 263us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:32:42.232892 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:42.234259 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:42.235859 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:42.236282 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:42.236403 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:42.237068 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:32:42.237547 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:32:42.237664 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:42.238163 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 11:32:42.238265 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":254,"threads_started":1}
I20260929 11:32:42.238890 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:42.230021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54484 (local address 127.23.153.61:38983)
0929 11:32:42.230286 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:42.230292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:42.231589 (+ 1297us) server_negotiation.cc:407] Connection header received
0929 11:32:42.232091 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:42.232098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:42.232200 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:42.232272 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:32:42.233273 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:42.234396 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:42.236756 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:42.237278 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:42.237395 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:42.237989 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:32:42.238120 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:32:42.238209 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:42.238366 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 11:32:42.238455 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260929 11:32:42.240339 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 11:32:42.231790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.23.153.60:40977)
0929 11:32:42.232003 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 11:32:42.232009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 11:32:42.233650 (+ 1641us) server_negotiation.cc:407] Connection header received
0929 11:32:42.233794 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 11:32:42.233798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 11:32:42.233868 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 11:32:42.233942 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 11:32:42.234483 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:42.235704 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:42.237831 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 11:32:42.238458 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 11:32:42.238578 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 11:32:42.239617 (+ 1039us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 11:32:42.239729 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 11:32:42.239822 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 11:32:42.239981 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 11:32:42.240067 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 11:32:42.243759 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 14520
I20260929 11:32:42.245311 14582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:42.366740 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 14520
I20260929 11:32:42.387727 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 14594
I20260929 11:32:42.388947 14656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:42.502560 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 14594
I20260929 11:32:42.520741 24164 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu pid 14669
I20260929 11:32:42.521955 14731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 11:32:42.631673 24164 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBSjPgi/build/asan/bin/kudu with pid 14669
2026-09-29T11:32:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4727 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9525 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (381435 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST